How Values work

Note - read this in conjunction with GUPPY.

Value is PCOT's fundamental type for working with uncertain numerical data. Values are triplets of:

  • a 32-bit floating point nominal value (i.e. a mean),
  • a 32-bit floating point uncertainty value (as standard deviation) around that mean,
  • and a set of 16 data quality (DQ) bits.

These are usually scalar, but 1D vectors can also be created.

As a user of PCOT you may never encounter Values, but they are used internally whenever any operations involving uncertainty are done. Here are the typical situations where that occurs:

Binary and unary operations on Value objects

This is the "base case". Values have dunder methods for operations. These are usually quite simple, although some (such as exponentiation) are nasty. They:

  • perform the operation to get the new nominal value. This is usually done with a lambda function, relying on Numpy's broadcasting to do the "right thing" with what it's given.
  • call a function to calculate the new uncertainty value
  • call a function to combine the two DQs
  • pass the three results into Value's constructor to get a new value

Examples - the multiplication operator:

    def __mul__(self, other):
        return Value(self.n * other.n, mul_unc(self.n, self.u, other.n, other.u),
                     combineDQs(self, other))

The AND operator:

    def __and__(self, other):
        """The & operator actually finds the minimum (Zadeh fuzzy op)"""
        n = np.where(self.n > other.n, other.n, self.n)
        u = np.where(self.n > other.n, other.u, self.u)
        d = np.where(self.n > other.n, other.dq, self.dq)
        return Value(n, u, d)

Binary and unary operations on Datum objects

This is for binops - unary operations are much the same, but a lot simpler (although imageUnop currently takes the underlying numpy array).

  1. Datum dunder function runs - this is usually very simple. For example, for addition it's just
    • return ops.binop(ops.Operator.ADD, self, other).
  2. Calls pcot.expressions.ops.binop with the appopriate ops.Operator code
  3. ops.binop runs
    • converting any raw numbers to Datum.NUMBER data with zero uncertainty.
    • Each possible triple (operator, leftdatum, rightdatum) of operator and Datum types will have a "binop semantics Datum wrapper" method registered for it by ops.initOps. For example, (multiplication, Datum.IMG, Datum.NUMBER) will call ops.imageNumberBinop with the lambda lambda x,y: x*y where the latter will take and return Value.
    • ops.binop calls the binop semantics method for the two Datum types
  4. For pairings of numbers and/or images, The binop semantics Datum wrapper converts any image data into a Value after performing subimage extraction, then calls the provided lambda. This calls the appropriate dunder method on the two Values. In fact, we could replace lambda x,y: x*y with Value.__mul__ etc., but it's much clearer this way.
    • see above for details of this.
  5. In imageNumberBinop (or equivalent) the returned array Value will be stitched back into the left-hand image using ImageCube.modifyWithSub, and that new image will be returned wrapped in a Datum. If the semantic wrapper was numberBinop, so that a number is returned, we just wrap that in a Datum.

For other types - not Datum.NUMBER or Datum.IMG - other semantics methods will have been written. For example, the ROI types have binary operators which construct ROIBinop objects using dunder methods:

    def regROIBinopSemantics(op, fn):
        """Used to register binops for ROIs, which support a subset of ops."""
        registerBinopSemantics(op, Datum.ROI, Datum.ROI, lambda dx, dy: ROIBinop(dx, dy, fn))

    regROIBinopSemantics(Operator.ADD, lambda x, y: x + y)
    regROIBinopSemantics(Operator.SUB, lambda x, y: x - y)
    regROIBinopSemantics(Operator.MUL, lambda x, y: x * y)
    regROIBinopSemantics(Operator.DIV, lambda x, y: x / y)
    regROIBinopSemantics(Operator.POW, lambda x, y: x ** y)

Binary and unary operations in an expr node

  • pcot.expressions.parse.InstOp runs, popping two Datum objects and calling a callback function stored in binopRegistry. This callback will have been registered by Parser.registerBinop, and is simply a lambda calling pcot.expressions.ops.binop with an Operator code for the operator, e.g.
    • p.registerBinop('*', 20, lambda a, b: binop(Operator.MUL, a, b)) (20 is the precedence)
  • The callback will call ops.binop, which is step 3 in the preceding section.

Again, unary operations are very similar but rather simpler.

functions of Value objects

Value generally has a method for each supported function. For example, the tan method will perform the trigonometric tan with uncertainty, returning a new Value. This is rarely called directly - instead, we generally work with Datum objects.

datumfuncs: functions of Datum objects

These are registered with the @datumfunc operator (see the plugins documentation) which can be found in pcot.expressions.register. This registers two separate wrappers: the expression wrapper, used when the function is called from an expr node; and the Python function wrapper, called when the function is called from inside Python.

They can take any number of arguments, but they must be either Datum objects or numeric.

Datumfuncs called from Python

The @datumfunc decorator will have wrapped the function in the Python decorator, so that subsequent calls from Python will go through the wrapper. This wrapper will

  • set default values for missing arguments (which must be string or numeric)
  • convert numeric and string arguments to Datum objects
  • call the original function and return the result (which must be Datum)

Datumfuncs called from expr nodes

The @datumfunc decorator also wraps the function in another wrapper - the exprwrapper - and registers the wrapped function with the expression parser. The wrapper itself is simpler than the Python wrapper because it can make use of facilities in the parser to convert values and check for errors. When data reaches the exprwrapper, we already know that the arguments are all Datum objects.

Datumfuncs of single numeric/scalar arguments

Functions of a single numeric or image Datum are sometimes written using the func_wrapper function in pcot.datum, which will turn imagecubes and numbers into Value objects in a similar way to how the semantic binop wrappers work for operations. It will also wrap the resulting Value in a Datum.

An example is Datum.sin, a method that finds the sine of a numeric or image Datum. Here is the method in full:

    def sin(self):
        return func_wrapper(lambda x: x.sin(), self)

The wrapper takes a function and the Datum on which perform the function. It returns a Datum. The function passed must take and return a Value. In this case, we use a lambda to call the Value class' sin method.

  • If the Datum is a Value (has type Datum.NUMBER), the wrapper will just call the function on the Value and wrap the returned value in a Datum.
  • If the Datum is an ImageCube (Datum.IMG), the wrapper will extract the mean, uncertainty and DQ arrays and put them into a Value and pass that to the function instead. If the returned value is a number, it will return that as a Datum. If it is an image it will splice it back into a copy of the original image (so that the function will only act within a region of interest if one is present) and return that.

We don't need worry about whether the argument is an image or a scalar, because the func_wrapper will deal with it. However, func_wrapper can only deal with images and scalars.

The datumfunc for sin simply delegates the operation to Datum:

@datumfunc
def sin(a):
    """
    Calculate sine of an angle in radians
    @param a:img,number:the angle (or image in which each pixel is a single angle)
    """
    return a.sin()

The stats wrapper

Another Datum wrapper which can be used is stats_wrapper. This wraps a function which takes a (mean,uncertainty,dq) tuple and returns another tuple of the same type. It does the following:

  • If provided with a numeric value, calls the wrapped function and creates a Value from the returned data.
  • If provided with an ImageCube, gets the subimage, splits it into bands, and calls the wrapped function on the non-BAD values in each band. The results for each band - assumed to be scalar - are converted into a vector Value with one value for each band.

This lets us write Datum's mean method like this:

def mean(self):
    return stats_wrapper(self,
        lambda n, u, d: (np.mean(n), pooled_sd(n, u), pcot.dq.NONE))

and it will work on scalar Values, vector Values and images - in the latter case, producing a vector Value.

The pooled_sd function will take the nominal and uncertainty arrays and pool their variation into a single value, as described in GUPPY.

The datumfunc is simple:

@datumfunc
def mean(val):
    """
    Find the mean and sd of a Datum. This does different things depending on what kind of Datum we are dealing with.
    For a scalar, it just returns the scalar. For a vector, it returns the mean and sd of the vector. For an
    image, it returns a vector of the means and sds of each channel.
    Pixels with "bad" DQ bits will be ignored.


    @param val:img,number:the value to process
    """
    return val.mean()