yex.parse.Source

yex.parse.Source(name=None)

A source is an iterator providing characters. A tokeniser reads characters from the source, and forms them into tokens.

The available subclasses handle:

• text files
• strings
• lists
• and null.

Attributes:

Name	Type	Description
name	typing.Union[str, None]	A name for this source, which appears in logs and error messages.
line_number	int	The line number (also called the row number). The first line is 1. If the line number is 0, we haven't started reading yet.
column	int	The column number. The first column is 1.
spin_check	int	How many times we've carried out a "read" operation without moving forwards. If this reaches Source.SPIN_LIMIT, then we throw SpinButStillError.
current_line	str	The contents of the current line we're working through. We keep the whole line until we're done with it, so that we can show it in error messages.
exhaust_at_eol	bool	If this is True, we act as though the end of the current line is the end of file. This is rarely needed: it's useful when we're reading input line by line from the terminal.
line_number_setter	typing.Union[typing.Callable, None]	If this is not None, we call it every time we begin a new line, with the line number as the single argument.
peeked	typing.List[str]	When someone uses our peek() method, we have to read the next character in order to know what to tell them. In order to maintain the illusion that the peeked character is still in the future, we push it onto this list. It follows that this list may only ever have zero or one members.
tail	str	The most recent ten characters (or fewer, if we haven't yet seen ten). This is only used by the "position" logger. Characters with a codepoint below 33 are represented by their counterparts in the Unicode block "Control Pictures", starting at U+2400. For example, a space is ␠.
lines	typing.List[str]	All the lines we've yet read from the file. The last entry in this list will be equal to self.current_line. This is wasteful, but useful sometimes. The list starts with a dummy blank entry, because lines in a file are counted from 1.
location	yex.parse.Location	Where we are in the file (or whatever), as a Location object. The class also provides properties for line and column numbers as ints, and the filename as a string.

Source code in yex/parse/source.py

def __init__(self,
             name = None,
             ):

    self.name = name
    self.column_number = 1
    self.line_number = 0
    self.current_line = ''
    self.spin_check = 0
    self.exhaust_at_eol = False
    self.line_number_setter = None
    self.peeked = []
    self.tail = ''
    self.lines = ['']
    self._iterator = self._read()

    logger.debug("%s: ready",
            self)

discard_rest_of_line()

Drops the whole of the rest of the current line.

Source code in yex/parse/source.py

def discard_rest_of_line(self) -> None:
    """
    Drops the whole of the rest of the current line.
    """
    if self._iterator is None:
        return

    if (
            self.column_number is not None and
            self.column_number != len(self.current_line)):
        logger.debug("%s: discarding the rest of the line (it was %s)",
                     self, repr(self.current_line[self.column_number:]))
    self._get_next_line()

peek()

Returns the character which our iterator will return next time. A glimpse into the future!

Source code in yex/parse/source.py

def peek(self) -> str:
    """
    Returns the character which our iterator will return next time.
    A glimpse into the future!
    """
    if not self.peeked:
        self.peeked.append(next(self))

    return self.peeked[0]