Anjuta token stream

## Anjuta token stream Anjuta token stream — Anjuta token stream |

## Stability Level

Unstable, unless otherwise indicated

## Synopsis

    #include <libanjuta/anjuta-token-stream.h>

    AnjutaTokenStream * anjuta_token_stream_push            (AnjutaTokenStream *parent,
                                                             AnjutaToken *root,
                                                             AnjutaToken *content,
                                                             GFile *file);
    AnjutaTokenStream * anjuta_token_stream_pop             (AnjutaTokenStream *stream);
    AnjutaTokenStream * anjuta_token_stream_get_parent      (AnjutaTokenStream *stream);
    AnjutaToken *       anjuta_token_stream_get_root        (AnjutaTokenStream *stream);
    GFile *             anjuta_token_stream_get_current_directory
                                                            (AnjutaTokenStream *stream);
    GFile *             anjuta_token_stream_get_current_file
                                                            (AnjutaTokenStream *stream);
    AnjutaToken *       anjuta_token_stream_tokenize        (AnjutaTokenStream *stream,
                                                             gint type,
                                                             gsize length);
    gint                anjuta_token_stream_read            (AnjutaTokenStream *stream,
                                                             gchar *buffer,
                                                             gsize max_size);
    void                anjuta_token_stream_append_token    (AnjutaTokenStream *stream,
                                                             AnjutaToken *token);
                        AnjutaTokenStream;

## Description

A AnjutaTokenStream object reads and writes a list of tokens. It uses two list. The first list is assigned when the object is created. Each token is read as characters discarding the separation between tokens. The second list is written using the data of the first list, so no new memory is allocated, in order to recreate a new list of tokens.

This is used when the lexer needs several passes. At the beginning the file is read as a single token containing the whole file content. The first pass split this content into tokens. Additional passes are done on some parts of the token list to get a more precise splitting.

It is important to not allocate new memory and keep the same character pointers in the additional passes because the token list does not own the memory. The address of each character is used to find the position of the changed data in the file.

Several objects can be linked together to create a stack. It is used for included file or variable expansion.

## Details

### anjuta_token_stream_push ()

    AnjutaTokenStream * anjuta_token_stream_push            (AnjutaTokenStream *parent,
                                                             AnjutaToken *root,
                                                             AnjutaToken *content,
                                                             GFile *file);

Create a new stream from a list of tokens. If a parent stream is passed, the new stream keep a link on it, so we can return it when the new stream will be destroyed.

`parent` : | a parent AnjutaTokenStream object or `NULL`. [allow-none]
`root` : | a token or `NULL`. [allow-none]
`content` : | a token list.
`file` : | a GFile of the file. [allow-none]
Returns : | The newly created stream.

### anjuta_token_stream_pop ()

    AnjutaTokenStream * anjuta_token_stream_pop             (AnjutaTokenStream *stream);

Destroy the stream object and return the parent stream if it exists.

`stream` : | a AnjutaTokenStream object.
Returns : | The parent stream or `NULL` if there is no parent. [transfer none][allow-none]

### anjuta_token_stream_get_parent ()

    AnjutaTokenStream * anjuta_token_stream_get_parent      (AnjutaTokenStream *stream);

Return the parent stream

`stream` : | a AnjutaTokenStream object.
Returns : | The parent stream or `NULL` if there is no parent. [transfer none][allow-none]

### anjuta_token_stream_get_root ()

    AnjutaToken *       anjuta_token_stream_get_root        (AnjutaTokenStream *stream);

Return the root token for the output stream.

`stream` : | a AnjutaTokenStream object.
Returns : | The output root token. [transfer none]

### anjuta_token_stream_get_current_directory ()

    GFile *             anjuta_token_stream_get_current_directory
                                                            (AnjutaTokenStream *stream);

Return the current directory.

`stream` : | a AnjutaTokenStream object.
Returns : | The current directory. [transfer none]

### anjuta_token_stream_get_current_file ()

    GFile *             anjuta_token_stream_get_current_file
                                                            (AnjutaTokenStream *stream);

Return the current file.

`stream` : | a AnjutaTokenStream object.
Returns : | The current file. [transfer none]

### anjuta_token_stream_tokenize ()

    AnjutaToken *       anjuta_token_stream_tokenize        (AnjutaTokenStream *stream,
                                                             gint type,
                                                             gsize length);

Create a token of type from the last length characters previously read and append it in the output stream. The characters are not copied in the output stream, the new token uses the same characters.

`stream` : | a AnjutaTokenStream object.
`type` : | a token type.
`length` : | the token length in character.
Returns : | The created token.

### anjuta_token_stream_read ()

    gint                anjuta_token_stream_read            (AnjutaTokenStream *stream,
                                                             gchar *buffer,
                                                             gsize max_size);

Read token from the input stream and write the content as a C string in the buffer passed as argument.

`stream` : | a AnjutaTokenStream object.
`buffer` : | a character buffer to fill with token data.
`max_size` : | the size of the buffer.
Returns : | The number of characters written in the buffer.

### anjuta_token_stream_append_token ()

    void                anjuta_token_stream_append_token    (AnjutaTokenStream *stream,
                                                             AnjutaToken *token);

Append an already existing token in the output stream.

`stream` : | a AnjutaTokenStream object.
`token` : | a AnjutaToken object.

### AnjutaTokenStream

    typedef struct _AnjutaTokenStream AnjutaTokenStream;