Allow sharing of ancillary variables among multiple data variables

[rhorne@…]

Currently it is not possible for multiple data variables to share an ancillary variable.

Augment the conventions to allow two or more data variables to share an ancillary variable.

Here is an example of a convention, which dovetails with the existing conventions, to provide this capability:

float swpt(lat,lon);

. .

swpt:standard_name="sea_water_potential_temperature"; swpt:ancillary_variables="nobs flags"; . .

float sws(lat,lon);

. . sws:standard_name="sea_water_salinity"; sws:ancillary_variables="nobs flags"; . .

int nobs(lat,lon);

. . nobs:standard_name="sea_water_potential_temperature sea_water_salinity number_of_observations"; . .

int flags(lat,lon);

. . flags:standard_name="sea_water_potential_temperature sea_water_salinity status_flag"; flags:flag_values = 0, 1, 2; flags:flag_meanings = "valid invalid unknown"; . .

Change the text in the CF metadata conventions section 3.3 that reads

"A standard name is associated with a variable via the attribute standard_name which takes a string value comprised of a standard name optionally followed by one or more blanks and a standard name modifier (a string value from Appendix C, Standard Name Modifiers)."

to

"A standard name is associated with a variable via the attribute standard_name which takes a string value that can have either of two forms. The first form is a standard name alone. The second form is a blank-separated list beginning with one or more standard names and ending with a single standard name modifier (i.e. standard_name [standard_name ...] standard_name_modifier). The standard name modifiers are defined in Appendix C, Standard Name Modifiers. This second form permits a single variable to provide ancillary data (see section 3.4) for two or more variables where each has a unique standard name."

Changes to CF software tools are required. For example, the CF_checker, which validates the standard_name attribute needs to change. However, it does not require any change to software that uses the complete attribute simply as an identifying string (e.g. to label plots, etc.)

[jonathan]

Many thanks for making this proposal. I think this extension to CF would be useful.

Jonathan

[ngalbraith]

There's no prohibition (that I can find!) in the current standard against an ancillary variable being shared by multiple data variables.

"The variables listed by the ancillary_variables attribute will OFTEN (my caps) have the standard name of the variable which points to them including a modifier"

To my mind, the problem is in the concept of standard name modifiers, and it would be worthwhile to find an alternate way to convey the information currently in those modifiers' definitions.

I'd like to see status_flag (and at least one of the other modifiers, number_of_observations) added as a standard name, possibly with a new attribute to indicate the standard names of the referring variables.

This seems less disruptive with respect to existing software, and more practical for systems where one ancillary variable is shared by many data variables.

[jonathan]

Dear Nan

There is no prohibition on pointing to the same ancillary variable from several data variables, but the difficulty is that if those data variables have standard names, it is not possible to choose a standard name for the ancillary variable! That's what this proposal aims to get round. You are right that we could make status_flag into a standard name in its own right. The drawback of that would be that it wouldn't so fully identify itself; it would not say what data it applied to. We could likewise do this with number_of_observations. Do you think that matters?

We could not do as you proposers with modifiers, such as standard_error, which have physical units, since those units depend on the data variable. Variables with different units must have different standard names.

Best wishes

Jonathan

[rhorne@…]

I volunteer to be the moderator for this trac item.

[ngalbraith]

Since the ancillary_variable attribute uses the short name, not the standard name, and since there is no prohibition against having 2 variables with the same standard name in a single file, you can also deal with shared ancillary variables like this:

float swpt(lat,lon);

swpt:standard_name="sea_water_potential_temperature"; swpt:ancillary_variables="swptnobs swptflags";

float sws(lat,lon);

sws:standard_name="sea_water_salinity"; sws:ancillary_variables="swsnobs swsflags";

int swptnobs(lat,lon);

nobs:standard_name="number_of_observations";

int swsnobs(lat,lon);

nobs:standard_name="number_of_observations";

The ancillary variable doesn't really need to point back to its "data variable" - the data variable points to the ancillary variable.

To be honest, I use a lot of ancillary variables that have no standard name, because this construct allows 2 dimensional meta data that does not fit into attributes; things like instrument descriptions (make/model/SN). So maybe this addition to the standard will not be a problem, but it seems to me to be fixing a problem that does not really exist.

[rhorne@…]

Hi Nan:

In the example you provided, neither of the two ancillary variable (i.e. ancillary data with names swptnobs and swsnobs) are shared by multiple data variables (i.e. the ones with names of swpt and sws).

Using your example, how can the data variables with the names of swpt and sws share the same ancillary variable (containing the number of observations) with the current set of CF conventions ?

The reason for this sharing would be to avoid duplicating ancillary data in the NetCDF dataset for performance and/or sizing reasons.

very respectfully,

randy

[ngalbraith]

You're right, I kept that short because I was just giving an example of why (IMHO) the standard name of an ancillary variable doesn't need to refer back to the "data variable" - since there can be multiple variables with the same standard name.

Here's how I'd code 2 data variables that share the nobs ancillary variable and each has its own flags ancillary variable. I just don't see any problem with this - the relationship between the data variables and the ancillary variables seems very clear.

float swpt(lat,lon);

swpt:standard_name="sea_water_potential_temperature";

swpt:ancillary_variables="nobs swptflags";

float sws(lat,lon);

sws:standard_name="sea_water_salinity";

sws:ancillary_variables="nobs swsflags";

int nobs(lat,lon);

nobs:standard_name="number_of_observations";

int swptflags(lat,lon);

swptflags:standard_name="flags";

int swsflags(lat,lon);

swsflags:standard_name="flags";

[rhorne@…]

Hi Nan:

Note that this enhancement is specifically intended to address the case when the ancillary variable is based on one of the modifiers in Table C.1, and its associated data variable has a standard_name.

Let me repost Jonathan's response from 2/24, because I think we are revisiting ground already discussed:

There is no prohibition on pointing to the same ancillary variable from several data variables, but the difficulty is that if those data variables have standard names, it is not possible to choose a standard name for the ancillary variable! That's what this proposal aims to get around. You are right that we could make status_flag into a standard name in its own right. The drawback of that would be that it wouldn't so fully identify itself; it would not say what data it applied to. We could likewise do this with number_of_observations. Do you think that matters?

We could not do as you propose with modifiers, such as standard_error, which have physical units, since those units depend on the data variable. Variables with different units must have different standard names.

very respectfully,

randy

[jonathan]

Dear Nan

The problem is illustrated by your example: you use number_of_observations and flags as standard names, but they are not in the standard name table, so that's not legal. A valid value for the standard_name attribute would be sea_water_potential_temperature number_of_observations, but it would be confusing to use a variable with that description as an ancillary for sea water salinity as well.

Your example proposes another way to solve this problem, which is to make number_of_observations a standard_name. It would not say what was being observed, though, so the ancillary variable would be less well-described in its own right. Does that matter? What do you or others think?

Best wishes

Jonathan

[mgschultz]

Hi Jonathan,

interesting thought. My initial reaction was "NEVER!", because such a generic name would contradict the principle of letting a standard_name describe as precisely as possible what a variable is supposed to mean. However, reading through the recent posts I can see the rationale for this. Yet, we would have to be careful to distinguish these generic names from the others that are normally assigned to a variable. Otherwise, everyone would use only "number_of_observations", when in fact "sea_water_potential_temperature number_of_observations" would be a lot more precise and indicate what we really want.

Perhaps it would be better to call this a "category_name" rather than a "standard_name"?

On the other hand: let's rewind a bit. If the standard_name were indeed the full "sea_water_potential_temperature number_of_observations", and another variable would point to nobs as well, wouldn't this then automatically suggest that the same number of observations were obtained for both variables? Does it matter that there is no variable with a standard_name "sea_water_salinity number_of_observations"? I think, this would be a cleaner approach without loss of functionality. It's up to the PI then to decide which of the two possible standard_names is to be used.

Martin

[ngalbraith]

I agree with Martin - 'sea_water_potential_temperature' could easily have an ancillary variable called 'sea_water_temperature number_of_observations.' (I'd tend to name these after observed variables and carry them over to calculated ones, but that should be left to the data writer.)

If we wanted to require that the NOBS variable name included the 'target' data variable name (e.g. sea_water_potential_temperature had to have a NOBS ancillary variable called 'sea_water_potential_temperature number_of_observations' or 'sea_water_temperature sea_water_potential_temperature number_of_observations' we wouldn't need to bother with the ancillary variable attribute at all.

I'm in favor of exploiting the flexibility this attribute gives us.

[rhorne@…]

Hi all !

Sorry for being negligent in bringing this trac item to closure. Let me recap where I think we are on this.

The most recently published version of the CF conventions (1.6), includes an “ancillary data” construct. This allows one data variable (a.k.a. the ancillary data variable) to provide metadata about the individual values of another data variable. To conform to the convention required for ancillary data requires:

(1) The use of an attribute with the name “ancillary_variables” for the data variable needing metadata to describe its elements. The attribute value is the name of the one or more associated ancillary data variables separated by spaces.

(2) Declaration of the ancillary data variable with the same dimension(s) and assignment of its standard_name attribute will often have the standard_name of the data variable needing metadata to describe its elements. In the standard name attribute of the ancillary data variable, a modifier is included to indicate the relationship (see Appendix C, Standard Name Modifier).

This current convention allows a single data variable to have multiple ancillary data variables, but does not allow multiple data variables to share a single ancillary data variable.

The proposal is to allow the standard_name attribute in the ancillary data variable to contain one OR MORE standard_names, which are associated with the data variables needing metadata to describe their elements,. (See the original proposal in the trac item.)

There was one counter-proposal presented that requires the standard name modifiers in Appendix C to have their own standard names and adding a new standard attribute to indicate the standard name of the referring variable, but this is problematic because these new standard_names do not fully identify themselves and the units could potentially vary, as is the case with the modifier standard_error.

There was another counter-proposal presented that requires a unique standard_name to be defined for the ancillary data variable where the unique name is a concatenation of the standard name of the data variable being referred to and an Appendix C standard_modifier. The problems with this approach are (1) the linkage from the ancillary data variable to more than one data variable is not provided, and (2) new standard_names would need to be defined whenever a data variable (having a defined standard_name) requires a new ancillary data variable, standard_modifier combination. This could cause a very large number of new standard_names to be defined. Also note that this approach defeats one feature of the existing ancillary data convention – no need to establish standard_names for ancillary data variables having standard_modifiers.

very respectfully,

randy

[rhorne@…]

Folks:

Almost a month has elapsed since I summarized the status of this trac item. In the summary, I advocated updating the CF conventions as identified in the original proposal above, which I will repeat here:

Change the text in the CF metadata conventions section 3.3 that reads

"A standard name is associated with a variable via the attribute standard_name which takes a string value comprised of a standard name optionally followed by one or more blanks and a standard name modifier (a string value from Appendix C, Standard Name Modifiers)."

to

"A standard name is associated with a variable via the attribute standard_name which takes a string value that can have either of two forms. The first form is a standard name alone. The second form is a blank-separated list beginning with one or more standard names and ending with a single standard name modifier (i.e. standard_name [standard_name ...] standard_name_modifier). The standard name modifiers are defined in Appendix C, Standard Name Modifiers. This second form permits a single variable to provide ancillary data (see section 3.4) for two or more variables where each has a unique standard name."

Note that...

Changes to CF software tools are required. For example, the CF_checker, which validates the standard_name attribute needs to change. However, it does not require any change to software that uses the complete attribute simply as an identifying string (e.g. to label plots, etc.)

No one has taken issue with the summarization for this trac item in the last 3 week.

I support this enhancement. Does the conventions committee support this enhancement ?

[russ]

I support this enhancement, except that I wonder if the last phrase in the last sentence could be misinterpreted to limit the use of this enhancement unnecessarily. Specifically, I would like to see the last sentence

This second form permits a single variable to provide ancillary data (see section 3.4) for two or more variables where each has a unique standard name.

clarified and simplified to say

This second form permits a single variable to provide ancillary data (see section 3.4) for variables that have a standard_name attribute.

This would permit multiple variables that have the same standard_name attribute to share the same ancillary variable, as could happen, for example, for different sensors measuring the same physical quantity at the same points for calibration or for sensors with overlapping ranges of application.

--Russ

[biard]

I'm afraid that I am confused by the change Russ has suggested. The second form is specifically intended for allowing the ancillary variable to explicitly express a "multiple parentage" relationship. Is there any reason why it would be acceptable to have an ancillary variable name multiple standard names if it is only referenced by a single primary?

[rhorne@…]

I think the confusion is a result of a mistake in the original revision to section 3.3 where I said:

"This second form permits a single variable to provide ancillary data (see section 3.4) for two or more variables where each has a unique standard name."

The statement should have been:

"This second form permits a single variable to provide ancillary data (see section 3.4) for one or more variables where each has a unique standard name."

The clarification/simplification Russ provides appears to be a good one.

The question posed by Jim:

Is there any reason why it would be acceptable to have an ancillary variable name multiple standard names if it is only referenced by a single primary?

The answer is no.

Here is the revised wording change to the CF standard as a result of Russ' recommendation:

Change the text in the CF metadata conventions section 3.3 that reads

"A standard name is associated with a variable via the attribute standard_name which takes a string value comprised of a standard name optionally followed by one or more blanks and a standard name modifier (a string value from Appendix C, Standard Name Modifiers)."

to

"A standard name is associated with a variable via the attribute standard_name which takes a string value that can have either of two forms. The first form is a standard name alone. The second form is a blank-separated list beginning with one or more standard names and ending with a single standard name modifier (i.e. standard_name [standard_name ...] standard_name_modifier). The standard name modifiers are defined in Appendix C, Standard Name Modifiers. This second form permits a single variable to provide ancillary data (see section 3.4) for variables that have a standard_name attribute.

[russ]

Jim,

My primary concern was a potential for misunderstanding the meaning of "unique" in the phrase "for one or more variables where each has a unique standard name". The standard_name attribute of any variable that has such an attribute is already unique, in the sense that a variable cannot have two attributes with the same name.

The only other interpretation of "unique standard name" I can think of would rule out multiple variables with the same standard_name attribute sharing the same ancillary variable. I don't see a good reason to rule out that case, and I provided an example where it might be useful.

The other minor change I suggested was omitting "two or more", because in fact the second form of the standard_name string value is also required to provide a standard name modifier for a single variable.

--Russ

[biard]

Russ,

I'm concerned that your proposed wording (and, to a lesser extent, Randy's wording) doesn't make it clear enough that this "two or more standard name" option is only to be used with a particular category of ancillary variables. A primary variable should never have more than one, should it?

I'm also concerned that the proposed wording can lead a reader into thinking that an ancillary variable must share a standard name with any primary variable that it has been associated with, which is not true.

I'll try to come up with my own version for your consideration.

Jim

[jonathan]

Dear Randy

Because of the recent concerns expressed in other discussions about standard name modifiers, I am curious to know which ones you actually use in practice. Is it the number_of_observations and the status_flag?

Best wishes

Jonathan

[rhorne@…]

Hi Jonathan:

We are using "status_flag".

Our algorithms that generate the gridded/imagery data sets also generate a quality value for each data point/pixel. There are cases where an algorithm outputs multiple gridded/imagery data sets and the same quality value applies to all the data sets.

very respectfully,

randy

[biard]

Randy,

Sorry it's taken so long to post on this again. I suggest that instead of modifying the second paragraph of section 3.3, we introduce a paragraph before the last sentence of the section that reads:

In the special case where more than one data variable references an ancillary data variable (see Section 3.4, "Ancillary Data") that contains a derived quantity (such as number of observations) that applies to all of the referencing data variables, and where the referencing data variables do not all share the same standard name, the standard_name attribute for the ancillary data variable is comprised of a blank-separated list of the unique set of standard names from the referencing data variables followed by the common standard name modifier.

Another way to resolve this question is to take an approach that is closer to Nan's suggestion. We could specify a special "placeholder" standard name - say, "?reference?", that would stand in for a specific standard name in the ancillary data variables standard_name attribute. The standard name table would have an entry for this new standard name that would explain that it is to be replaced by the standard name of a referencing data variable. This would allow us to keep the syntax of the standard_name attribute simple, avoid the headless standard name modifier problem, make it obvious that the contents of the ancillary data variable are applicable to one or more referencing data variables, and (eventually) lead to less confusion on the part of users than I think the multiple standard names approach may produce.

[rhorne@…]

Jim:

Thanks for the inputs.

Let me first address the counter-proposal. There is no precedence (yet) for reserved keywords in the list of standard_names. Given the fact that there is a less disruptive option that leverages the thinking behind the existing convention, I suspect the message board and committee would find this option less appealing.

As far as your rewording of the existing proposal, I can see that your intent is to make the language more precise to avoid confusion. The fact that you got confused means that there is some likelihood that others will as well. I think your language could be further improved to make it clear that the current proposal does not apply to all of the standard name modifiers in Appendix C. In addition, I think that placing this rewording near the end of paragraph 3.3 increases the likelihood that it will be missed. As a result, here is the revised proposal:

Right after this existing statement of:

"A standard name is associated with a variable via the attribute standard_name which takes a string value comprised of a standard name optionally followed by one or more blanks and a standard name modifier (a string value from Appendix C, Standard Name Modifiers)."

Include the following statement in the same paragraph:

"In the case where more than one data variable, each with a standard name, references an ancillary data variable (see Section 3.4, “Ancillary Data”) for the same reason (e.g. status_flag), the standard_name attribute for the ancillary data variable is composed of a blank-separated list of each unique standard name followed by one or more blanks and a standard name modifier."

The ambiguity that remains in the statement (i.e. the case where multiple data variables referencing the ancillary data variable have the same standard_name) is moot. As a result, there is no need to further increase the complexity of the statement to resolve it.

Does this pass muster with the convetions committee ?

very respectfully,

randy

[mgschultz]

Dear all,

I am all for this proposal. Only, I would like to raise one more technical point, i.e. we should define the proper way for parsing a standard_name with this extension. If I am not mistaken, then the current proposal mandates to read a standard_name attribute from right to left, checking first if the rightmost part is actually a standard_name modifier. If so, then one or more standard_names (separated by blanks) are allowed to the left. If the rightmost string is no standard_name modifier, it is supposed to be a standard_name and must be one string (i.e. no blanks).

For a moment, I was wondering whether it would make sense to think a little broader and introduce the concept of a "standard_name pointer", meaning that - through a modifier - the variable content is no longer referring to a quantity that is described by the standard_name itself, but the standard_names that are listed, are merely references to variables with that content. It may then be useful to indicate this explicitly in the syntax, for example, by prepending a "*" as in the C language, or a more xml-style identifier such as "X:". The example given at the top of this thread would then become:

int nobs(lat,lon);

    . . nobs:standard_name="X:sea_water_potential_temperature sea_water_salinity number_of_observations"; . .

... and then it would probably be easier to understand if the modifier would come first, followed by the list of variables to which it refers, i.e.:

int nobs(lat,lon);

    . . nobs:standard_name="X:number_of_observations sea_water_potential_temperature sea_water_salinity"; . .

I realize, this is a more radical change. If there are sympathies for this, then we should perhaps schedule it for CF 2.0?

Best,

Martin

[rhorne@…]

Martin:

I can not speak for all, but from posting and reading board and trac messages for a while, there is a resistance to the use of software-language-like conventions. Personally, I believe this overarching thinking has served the standard and the community that uses the standard well.

For this particular enhancement, while the complexity of the software parsing the standard_name attribute is increased, the new syntax is a natural extension of the exising convention and likely to minimize confusion.

For these reasons, I think the approach and latest verbage change to the CF standard proposed is a conservative and reasonable path forward.

I will include the change to the CF standard in this post for completeness:

In paragraph 3.3 Standard Names, right after this existing statement of:

"A standard name is associated with a variable via the attribute standard_name which takes a string value comprised of a standard name optionally followed by one or more blanks and a standard name modifier (a string value from Appendix C, Standard Name Modifiers)."

Include the following statement in the same paragraph:

"In the case where more than one data variable, each with a standard name, references an ancillary data variable (see Section 3.4, “Ancillary Data”) for the same reason (e.g. status_flag), the standard_name attribute for the ancillary data variable is composed of a blank-separated list of each unique standard name followed by one or more blanks and a standard name modifier."

very respectfully,

randy

[ngalbraith]

Replying to rhorne@excaliburlabs.com:

"In the case where more than one data variable, each with a standard name, references an ancillary data variable (see Section 3.4, “Ancillary Data”) for the same reason (e.g. status_flag), the standard_name attribute for the ancillary data variable is composed of a blank-separated list of each unique standard name followed by one or more blanks and a standard name modifier."

I really seem to be out of step, but honestly I still don't see what we're gaining here. More important, I think we're making assumptions about the role of a standard name in an ancillary variable that may lead to problems sharing data.

IMHO, in a file with multiple variables that share one or more ancillary variables, there is no need - and no really reliable method - for the ancillary variables to list all their parents. This is a one-way connection; a primary variable points to its ancillary variables, using short names.

Is there a use case for ancillary variables to identify their primary variables? If we actually need a 2-way system, is there a sense that this method - listing all the primaries' standard names - will work?

What happens when there are multiple 'primary' variables with the same standard name in a file?

What's wrong with this:

 float swt(time,depth,lat,lon);
    swt:standard_name="sea_water_temperature";
    swt:ancillary_variables="swt_nobs";
 float swpt(time,depth,lat,lon);
    swpt:standard_name="sea_water_potential_temperature";
    swpt:ancillary_variables="swt_nobs";
 int swt_nobs(time,depth);
    swt_nobs:standard_name="sea_water_temperature number_of_observations";

[jonathan]

Dear Randy, Nan et al.

I think the difference between Randy's proposal and Nan's comment highlights a different perception about the purpose of standard_names and modifiers. I believe the purpose of the standard name is to make the data variables self-describing, and the modifiers were introduced as an extension to support this purpose without expanding the standard name table too much. If that's the purpose, Randy's proposal is logical because the data variable for status flag or number of observations variable needs to describe itself as applying to various different quantities. This would be particularly helpful if the status flag data variable was stored in a different file from the primary data variables, for example, in which case it would lose the association if it could not describe itself. Thus, I don't think the modifiers are principally intended as pointers.

On the other hand, the ancillary_variables attribute was deliberately introduced as a pointer. It makes life easier by pointing from the primary data variable to its metadata variables. It could be omitted, because the association can be inferred from the standard names. In Nan's example, if you scan the file and find a variable with standard_name of sea_water_temperature number_of_observations, you know that it is a metadata variable which belongs to a primary data variable with a standard_name of sea_water_temperature; the ancillary_variables attribute of the primary data variable is a convenience feature which saves your having to scan the file.

I would suggest that the above is an answer to Nan's comment, and supports Randy's proposal. However, there have been other threads discussing standard name modifiers, which together give the impression that they are confusing and/or were ill-conceived. A particular restriction about the interpretation I have given and Randy's proposal is that it can only apply when the metadata variables for the various primary data variables have the same units. That is OK for status_flag and number_of_observations, which are dimensionless. Metadata variables with these modifiers could apply, for example, to both sea_water_temperature and sea_water_salinity. A metadata variable with the standard_error modifier could not, because it would have a different unit for the two primary data variables.

Therefore I am wondering about a simpler solution than Randy's or Martin's. Maybe we should introduce status_flag and number_of_observations as standard names in their own right. The drawback with this would of course be that the metadata variables with these standard names would not be so completely self-described. You would not be able to tell which primary data variable they belonged to if they were in different files. Within the same file, you'd have to rely on the ancillary_variables pointer if there was any ambiguity. But maybe this would not be too bad. There are plenty of other relationships between data variables in files that are application-specific and not indicated by any CF metadata e.g. it's a reasonable assumption that sea_water_temperature, sea_water_salinity and sea_water_density are all physically consistent.

What do you think? This would not be a solution to the other existing modifiers of standard_error and detection_minimum, which are not dimensionless, but other discussions have suggested that those could be replaced with cell_methods. That would be a topic for a different ticket.

Best wishes

Jonathan

[biard]

Jonathan,

Your last comment has helped me understand (and agree with) Nan's objection.

As near as I can tell, a standard name is a sort of type or class. It describes the contents of a variable by telling you what class of measurement it belongs to, but I think its reliable use must end there.

If two variables share a standard name (or share a root standard name), it tells you about comparability, but it doesn't reliably tell you anything about linkage. The only situation in which it might be safe to infer linkage from standard names is when there is only one primary variable in a file with a given standard name, and in which all of the ancillary variables for that primary variable have standard names that are modifications of the primary. There are a number of conditions that invalidate this approach. For example:

1) More than one primary variable with the same standard name. (Which each need their own ancillary variables.) 2) A primary variable that has a standard name that is a modification of the standard name associated with another primary variable. 3) An ancillary variable has the same standard name as its primary. 3) An ancillary variable has a standard name that is completely different from its primary.

It seems pretty clear to me that the ancillary_variables attribute is not a convenience feature. It is the only general way available in CF to unambiguously declare the linkages.

Given all that, I don't have any problem with promoting status_flag and number_of_observations to the level of standard names.

Grace and peace,

Jim

[graybeal]

Jim wrote:

"As near as I can tell, a standard name is a sort of type or class. It describes the contents of a variable by telling you what class of measurement it belongs to, but I think its reliable use must end there. If two variables share a standard name (or share a root standard name), it tells you about comparability, but it doesn't reliably tell you anything about linkage. ... Given all that, I don't have any problem with promoting status_flag and number_of_observations to the level of standard names."

I agree with everything written above (and in the original post), except the last sentence. The motivation is not that the rest of the standard name provides traceability, it is to provide better descriptive information about the parameter (and perhaps its comparability) -- as was mentioned much earlier in this thread. (Is there ever a status flag that isn't a status flag about some measurement? Is there ever a number_of_observations that isn't a number of observations of something?)

If the proposal is to allow appending _status_flag to a standard name, that would be OK, but not status_flag by itself (unless we require the relevant auxiliary information to be provided some other way).

[jonathan]

Dear John

The modifiers are effectively a suffix, like appending status_flag to a standard name, without having to define a new standard name. I am suggesting, indeed (as Jim says), that status_flag should be defined as a standard name in its own right. Of course, it must really be the status flag of something, but this would not be stated. The cost is therefore loss of descriptive information, as you say. The benefit is that it would resolve this rather awkward problem of primary data variables sharing metadata variables. Is the cost worth the benefit?

Cheers

Jonathan

[graybeal]

Definitions for the proposed new standard names might clear up confusion at this point.

Sorry, I didn't read far enough back in the thread with enough care. The rewording that left us with "In the case where more than one data variable, each with a standard name, references an ancillary data variable (see Section 3.4, “Ancillary Data”) for the same reason (e.g. status_flag)," created an uncertainty for me, namely: Are they really sharing metadata variables in the sense of the _same_ value applying to multiple primary variables? (As opposed to, say, allocating bits or something....)

I see the answer is yes, and yet I still couldn't fathom the use case at first. So is the proposal now to eliminate the proposed text about ancillary variables, instead allowing these generic standard names as the complete answer? Or still to include the proposed text for ancillary variables? (In which case, can I propose instead the following wording :"In the case where the values of an ancillary data variable (see Section 3.4, "Ancillary Data") apply to multiple data variables simultaneously, the standard_name attribute..." ?)

I withdraw my absolute reservation about promoting these to standard names, but assume it will be required to specify the variables to which the-variable-with-one-of-these-standard-names-applies. (Geez, I know that's unparseable, but can't fix it...) Otherwise you have a free-floating standard name with no effective means to understand what it really means.

[rhorne@…]

Folks:

If we go with this standard_name = "status_flag" approach, will the same approach be used for the other standard_name modifiers ?

An advantage of doing this for status_flag and the other standard name modifiers that has not yet been identified is that the data variable requiring the quality/status variable does not need a standard name.

very rezpectfully,

randy

[jonathan]

Dear Randy

I would suggest doing this only for status_flag and number_of_observations, which are dimensionless. The others could potentially be described by cell_methods, but that would be the subject for another ticket.

It would not be an improvement in metadata to omit the standard_name for the data variable. The standard_name is always an optional attribute, but it should be included whenever possible, I would say. However, I see your point.

Best wishes

Jonathan

[rhorne@…]

Folks:

I think this is where we are:

(1) staus_flag and number_of_observations become a special type of standard_name, and become the standard name of the applicable ancillary variable.

(2) these special standard_names are associated with ancillary variables that provide status or number of observations for two or more other data variables in the same data set.

(3) these other data variables must include the "ancillary_variables" attribute set to the variable name of the relevant ancillary variable.

Has converge of an approach to allow sharing of a status_flag or number_of_observation ancillary variable been reached ?

One other question that comes to mind ...

Is this syntax used only when an ancillary variable is associated with more than one data variable ?

very respectfully,

randy

[ngalbraith]

Is this syntax used only when an ancillary variable is associated with more than one data variable

That would be difficult or impossible to enforce, since so many data files can be split up on the fly via thredds and other services.

Also, we might want to consider allowing all 4 of the existing modifiers to have standard names (or category names, as was suggested earlier), as long as they are applied to primary data variables that share units. There could easily be a single detection_minimum, and maybe also a single standard_error, for e.g. wind or current vectors.

[jonathan]

Dear Nan

Replying to ngalbraith:

Also, we might want to consider allowing all 4 of the existing modifiers to have standard names (or category names, as was suggested earlier), as long as they are applied to primary data variables that share units. There could easily be a single detection_minimum, and maybe also a single standard_error, for e.g. wind or current vectors.

I don't think we can do that, because a given standard name must have unique canonical units. We couldn't define the canonical unit for detection_minimum or standard_error, since they depend on the geophysical quantity, as you imply.

Best wishes

Jonathan

[rhorne@…]

Folks:

The most recent approach to providing a CF compliant construct for allowing data variables to share ancillary variables discussed in this ticket elevates number_of_observations and status_flag to standard_names. One of these standard_names are assigned to the applicable ancillary variable, and the multiple data variables that have the designated relationship with the ancillary variable indicate so with an ancillary_variables attribute assigned to the variable name of the shared ancillary variable.

The most recent question surrounding the most recent approach to providing a CF compliant construct for allowing data variables to share ancillary variables revolve around whether all four current standard_name modifiers current defined in appendix C (detection_minimum, number_of_observations, standard_error, & status_flag) can make use of this extension.

Only the standard_name modifiers with dimensionless units (number_of_observations, states_flag) make sense because the other two standard_name modifiers, detection_minimum and standard_error, need to have the same units as the data variable. For multiple data variables needing to share the same ancillary variable, it would be overly restrictive to require they have the same units.

Has consensus been reached. ?

very respectfully,

randy

[ngalbraith]

I don't think it's so much about being too restrictive; if wind east and north share a detection_minimum there would be no problem using the same units for all 3 variables (east, north, and detection) and this could be left to the data writer to ensure sensible groupings.

The problem comes with the standard name table, where there's no way to assign canonical units to a variable named detection_minimum, which would need to take its units from the data variable to which it was attached.

[graybeal]

The problem brings its own solution. it should be fine to let multiple variables share the same ancillary variable, if it makes sense in all respects. For it to make sense when the standard_name needs to have the units of the data variable, all the variables need to have the same units.

Looks to me like it makes sense for detection minimum to allow this, in some environments it would be a common use case. But I can't figure out a sensible use case for one standard_error variable for multiple variables!

[ngalbraith]

Replying to graybeal:

The problem brings its own solution. it should be fine to let multiple variables share the same ancillary variable, if it makes sense in all respects. For it to make sense when the standard_name needs to have the units of the data variable, all the variables need to have the same units.

Hmm, I don't think there's been any question of multiple variables sharing a single ancillary variable; that's already permitted in the standard. We're now at the stage of deciding whether the current standard name modifiers should be elevated to free-standing standard names, so that these variables no longer carry the target variable(s) as part of the standard_name attribute.

The problem now is that standard names have specific units assigned to them, and this does not work for variables like standard_error or detection_minimum, which take their units from their 'data variables'. We would need to change the structure of the standard name table to accommodate this type of name - I'm not sure we have the motivation to do that.

Looks to me like it makes sense for detection minimum to allow this, in some environments it would be a common use case. But I can't figure out a sensible use case for one standard_error variable for multiple variables!

Agreed, more or less; however standard_error could still be a stand-alone standard name, so that the concept of the standard name modifier could be retired. The idea here is that the data variable points to the correct standard_error, using the ancillary_variables attribute, and the error variable does not include the standard name of the data variable in its standard_name attribute.

float swt(time,depth,lat,lon);
    swt:standard_name="sea_water_temperature";
    swt:ancillary_variables="swt_std_err";
float swt_std_err(time,depth,lat,lon);
    swt_std_err:standard_name="standard_error";
    swt_std_err:long_name="standard error of sea water temperature"

This makes the standard error variable a little less independent - you would not know what it is without looking at other variables in the file. To me, this is not an issue, but it might be important to other people.

On the other hand, the current method, where swt_std_err has the standard name 'sea_water_temperature standard_error', does not work when there are multiple variables with a standard name sea_water_temperature in a file.

[jonathan]

Dear all

I tend to think, partly as a result of discussion in another ticket, that we should adopt a different solution for number_of_observations and status_flag on the one hand, and standard_error and detection_minimum on the other. Thus we could get rid of standard name modifiers, as Nan says; they are awkward and have caused confusion.

As Randy says, the former two modifiers could become standard names for dimensionless quantities. As Nan says, this can't be done for the latter two, because they have units. Instead, I think we should put the information into cell_methods. It is possible to regard standard error and detection minimum as particular statistics in an ensemble of possible measurements of the same quantity, I would argue. If so, cell_methods is a natural place to put them, under an entry which applies to a notional dimension that runs over the members of this population. You could describe a standard error with a cell_methods entry of measurement: standard_deviation, for instance, if we add measurement to the standard name table, meaning the ordinal number of a measurement of a given quantity. There does not have to a measurement dimension; using a standard name in cell_methods implies that the statistic applies to the complete range of the quantity named i.e. all possible measurements in this case. Does this make sense to you?

A nice consequence is that this would simplify the convention, because only the standard name and cell methods would determine the canonical units of the quantity. At the moment, the standard name modifier has to be considered as well.

Cheers

Jonathan

[jonathan]

Dear Randy

On the email list you asked for an example to explain what I mean. I can't think of an example as such. Would you agree that the standard error of a measured quantity means, in principle, the standard deviation of the values you would get by repeatedly measuring the quantity under the same conditions (as far as you can control the conditions)? I believe that's the idea people have wanted to convey by this modifier. The values are not all exactly the same because of uncontrollable variations in the conditions and the behaviour of the instruments and methods being used to make the measurement. If you did in fact make many measurements and wanted to record them in a CF-netCDF file, it would be natural to have a dimension with an index number (1, 2, 3, ...) running over the values you got. A possible standard name for the coordinate variable containing this index could be measurement e.g.

dimensions:
  measurement=10;
variables:
  float temp(measurement);
    temp:standard_name="sea_water_temperature";
    temp:units="K";
  int measurement(measurement);

If you calculate the mean or the standard deviation of the measurements, you would use cell_methods to record this operation:

dimensions:
  measurement=1;
variables:
  float temp(measurement);
    temp:standard_name="sea_water_temperature";
    temp:units="K";
    temp:cell_methods="measurement: standard_deviation";

It is allowed to omit the dimension and include instead the standard name of the dimension in cell_methods. See Section 7.3.4. So we can have "measurement: standard_deviation" in cell_methods even if there is no measurement dimension. I think we could take that to imply the standard deviation of all possible values this measurement could take, which is what is meant (I think) by the standard error of the measurement. This might be stretching 7.3.4 a bit, and if so, an extra sentence could be inserted to explain this usage of the convention.

Is that any clearer?

Cheers

Jonathan

[rhorne@…]

Dear Jonathan:

I think I understand where you are going. So, when you say "It is allowed to omit the dimension and include instead the standard name of the dimension in cell_methods.", would the syntax for the variable containing the standard deviation of temperature look like the following (I took the liberty of changing the name of the standard deviation variable and I am assuming the standard_name in your example for "measurement" is "depth")?

variables:

float std_dev_temp;

temp:standard_name="sea_water_temperature";

temp:units="K";

temp:cell_methods="depth: standard_deviation";

Also, is it correct to assume there is an ancillary_variables attribute set to "std_dev_temp" in the temp variable ?

very respectfully,

randy

[biard]

So, now we are back to where we were before. Part of this proposal of Jonathan's still gives me heartburn. Expressed this way, the only way to know (apart from having it stated in the variable name, which is unreliable) that the variable contains a standard deviation of temperature and not temperature is to reference the cell_methods attribute.

The word "measurement" itself would become a standard name. It would not be a proxy or placeholder. It would have a status equivalent to longitude or time, and represent an ordinal counting of samples. Another similar example is the use of the word "area" in cell methods. Even though area may not ever be expressed directly in any variable, you can validly state in the cell_methods attribute that the variable values represent an average over the area of the cell.

Thus (if I understand correctly), "measurement: standard deviation" is stating that the operation "calculate the standard deviation" was performed on all the individual measurements that fell in this cell. I think it might be more restrictively understood as stating that the only dimension over which the operation was performed was the measurement dimension, and all other dimensions (lat and lon, for example) were held constant.

Jonathan, I'm not sure how or why you shifted from the dimension "measurement" having a size greater than one to having a size equal to one in your example. That bit just confused me.

If I am close to on target, the suggested ensemble of variables would be something like this.

dimensions:

time=N;

variables:

int time(time);

...

float a(time);

a:long_name="This is the average temperature measurement"; a:standard_name="sea_water_temperature"; a:units="K"; a:cell_methods="measurement: mean"; a:ancillary_variables="b";

float b(time);

b:long_name="This is the standard deviation of the average temperature measurement"; b:standard_name="sea_water_temperature"; b:units="K"; b:cell_methods="measurement: standard_deviation";

Is that about right, Jonathan?

Grace and peace,

Jim

[ngalbraith]

Replying to jonathan:

As much as I'd like to be able to get rid of standard name modifiers, I think they're far more clear than cell methods for concepts like standard_error and detection_minimum. When the quantity in the data variable is not representative of the concept expressed by the standard name, you're likely to have data misinterpreted; a standard name modifier is much harder to miss than a cell method.

Yes, we should all be looking at cell methods, but most of the method terms convey only ancillary information, at least for in situ data sets. The whole concept of cell methods could be improved if they were limited to descriptions of sampling/recording strategies, rather than including terms that transform the data variable from a geophysical phenomenon to a statistical value.

This is probably well out of the scope of this ticket, but maybe we should consider moving standard_deviation and variance from cell methods to standard name modifiers, since data variables with these 'methods' do not represent the the geophysical quantity expressed by the standard names.

Are there cases where these are used with a single axis, which would make that impossible? It seems more logical to me to group these with standard error and detection minimum than with point/max/mean/min etc.

[jonathan]

Dear all

Jim's description of what I meant is correct. The original dimension is replaced by a dimension of size one when you collapse it by calculating the statistic. In just the same way, a variable which is a function of longitude, having a longitude dimension of size greater than one, becomes a variable with a longitude dimension of size one when you calculate the zonal mean of it. We record the operation of calculating the zonal mean in cell_methods with an entry longitude: mean, where longitude is the name of the longitude dimension (now of size one). Another example, closer to my suggestion of measurement, is a model ensemble. It is possible to have a dimension for ensemble member. It is a common operation to calculate ensemble mean and ensemble standard deviation, and we can record these in cell_methods in the same way.

I respectfully disagree with Nan about the importance of cell_methods. The standard name does not contain the entire description of the data variable. I think it's important to keep in mind that CF metadata is distributed over several attributes, which should always all be considered. The cell_methods makes important distinctions about how the values represent variation over coordinates. Its whole purpose is to represent statistical operations, and that's why I've made the suggestion in this ticket, because there seems to be an analogy to me. There is always a statistical operation involved in calculating a value, unless it's truly a point value. Moreover, cell_methods is a more flexible attribute than the standard name because it refers to the dimensions, which may have been treated statistically in different ways: for instance, a quantity might be a time-mean and a zonal maximum, or it might be (to address Nan's question) an interannual standard deviation of monthly means (time: mean within years time: standard_deviation over years in cell_methods).

Best wishes

Jonathan

[biard]

Hi.

I agree with Nan. It sure seems to me that the standard_name for a variable is intended to indicate a class of measurements to which a variable belongs, such as sea_water_temperature, in order to identify variables that have roughly comparable contents. Requiring decoding of the cell_methods attribute in order to tell if two variables are actually comparable appears to directly violate what I understand to be the intent of the standard_name attribute. With the approach suggested by Jonathan, the standard_name is reduced to making a much fuzzier statement - effectively, "This variable has something other to do with measurements of X".

Maybe I'm wrong about the original/current interpretation of the standard_name attribute. If its purpose was always fuzzier than I thought it was, then let's move on. But if we are, in fact, talking about making a significant change in its interpretation, then I think we need to take a different approach.

Grace and peace,

Jim

[jonathan]

Dear Jim

We are not talking about a change of intention or interpretation of either attribute. cell_methods and standard_name are both original parts of the CF convention, and cell_methods has always included standard_deviation, variance, minimum and maximum as well as point and mean, for instance. The standard name is not the whole description. For instance, a commonly discussed quantity is the daily maximum of screen-height temperature. In CF, this is described by standard_name of air_temperature, cell_methods of time: maximum, and a height coordinate of 2.0 m (or 1.5 m, or whatever it was). It has been recognised in other discussions that there may also be a need to identify groups of metadata like this which together characterise a quantity. That was discussed under the "common concepts" trac ticket, which wasn't concluded, and more recently a method has been proposed in the ​CF standard string ticket. It would be nice to conclude that one!

However, in this ticket, if my suggestion that standard error of measurement is analogous to standard deviation in other dimensions doesn't seem natural to anyone else, it's obviously not the right approach to take. I was hoping to get rid of standard_name modifiers entirely, but we don't have to if we can't devise a better way. We could go back to the proposal as Randy had it at around ​comment 33, on which we largely agreed, I think, which solves the problem for Randy's applications and would leave standard_error as a modifier.

Cheers

Jonathan

[rhorne@…]

Dear All:

The message is a recap and a suggestion for a path forward.

The intent of this trac item is to provide a convention that allows a status_flag variable to be associated with more than one data variable. In the project I am working, the data variables have different standard_names, and, for the enhancement to be useful to us, it needs to support this case.

In the declaration of the status flag variable, the current conventions allow only a single standard name, which is that associated with the data variable, followed by the standard_name modifier “status_flag” to be the standard_name value.

The initial proposal to allow multiple data variables, each with a different standard_name, to be associated with a single standard_name included multiple, space-delimited standard_names to exist as values in the status flag variable’s standard_name attribute followed by "status_flag".

A subsequent proposal involved promoting two of the current standard_name modifiers, status_flag and number_of_observations to standard_name status. This allows multiple data variables to share status_flag or number_of_observation variables per the following:

float data_1(lat,lon); data_1:ancillary_variables="status_1"; float data_2 (lat,lon); data_2:ancillary_variables="status_1"; int status_1 status_1:standard_name="status_flag";

Note that the reason that only status_flag and number_of_observations could use this approach is that they are unitless versus the other two current standard_name modifiers, standard_error and detection_minimum that have the same units as the related data variable.

The con of this approach was identified, and it revolves around the standard_names “status_flag” and “number_of_observations” not being interpretable without understanding the related data variables.

There was then some discussion that ensued related to how the functionality provided the standard_name modifiers, standard_error and detection_minimum, could be subsumed with cell_methods. This discussion dovetailed with that which occurred on message thread “[CF-metadata] Question from NODC about interplay of standard name modifiers, cell_methods, etc.”

While I think there is a strong argument to be made that standard_error and detection_minimum are essentially cell_methods, I would suggest we work this issue, separately, in the context of a different discussion topic / trac item.

It would seem at this juncture that the reasonable candidate solutions to allowing multiple data variables to share a status_flag variable have been explored sufficiently well. It is pretty clear there is no perfect solution.

It would also seem that there is some validity to the argument revolving around the overlap in functionality provided by cell_methods, and the standard_error and detection_minimum standard_name modifiers.

As a result, I think the best option to allow multiple data variables to share a status_flag variable is to promote status_flag and number_of_observations to standard_name status as described above. These would no longer be standard_name modifiers. Going with approach sets the stage for subsequent discussion about cell methods and the remaining two standard_name modifiers.

Is this a reasonable path forward ? If so, I will write up the changes to the standard in a subsequent post.

very respectfully,

randy

[jonathan]

Dear Randy

Thank you for the helpful summary. I think this is a reasonable approach i.e. add status_flag and number_of_observations as standard names of dimensionless quantities. Regarding these, I would note:

• Although the standard name of the ancillary variable will no longer identify the quantity it belongs to, the ancillary_variables attribute serves to link them.

• So far, we have not abolished any existing facility of CF. We do not need to withdraw status_flag or number_of_observations as standard names. We could deprecate them i.e. recommend against using them. In that case the CF checker would give a warning (rather than an error) if they were used in a file conforming to the version of CF in which this change was made. I suggest that this option should be considered.

A separate ticket can deal with any proposal for standard_error and detection_minimum, as you suggest.

Cheers

Jonathan

[ngalbraith]

I agree, this is a clear summary; thank you, Randy.

Replying to jonathan:

• So far, we have not abolished any existing facility of CF. We do not need to withdraw status_flag or number_of_observations as standard names. We could deprecate them i.e. recommend against using them.

I think the above should be amended to read do not need to withdraw status_flag or number_of_observations as standard name modifiers, unless I'm completely missing the gist of the decision.

[jonathan]

Quite right, thanks for the correction, Nan. Jonathan

[rhorne@…]

Dear All:

This post defines "strawman" redlines to the CF standard to reflect this enhancement in the CF standard.

(1) In paragraph 3.5 Flags ... Change example 3.3 to be:

byte current_speed_qc(time, depth, lat, lon) ;

current_speed_qc:long_name = "Current Speed Quality" ; current_speed_qc:standard_name = "status_flag" ; current_speed_qc:_FillValue = -128b ; current_speed_qc:valid_range = 0b, 2b ; current_speed_qc:flag_values = 0b, 1b, 2b ; current_speed_qc:flag_meanings = "quality_good sensor_nonfunctional outside_valid_range" ;

The only change made to this example is the removal of the "sea_water_speed" from the flag variable's standard_name.

Immediately after the example, add the following sentence:

"Note that the data variable containing current speed has an ancillary_variables attribute with a value containing current_speed_qc."

(2) In Appendix C. Standard Name Modifiers ...

Add the following statement after the existing description for the number_of_observations modifier.

"The use of this modifier is deprecated and the standard_name number_of_observations is preferred to describe this type of metadata variable."

Add the following statement after the existing description for the status_flag modifier.

"The use of this modifier is deprecated and the standard_name status_flag is preferred to describe this type of metadata variable."

(3) Note that a post requesting standard_names status_flag and number_of_observations are posted to the CF metadata message board.

very respectfully,

randy

[jonathan]

Dear Randy

Thanks. I think that is fine and I support this change being made. We should also add to Sect 3.3 of the conformance document:

Recommendations:

• Use of the standard_name modifiers status_flag and number_of_observations is deprecated, and the corresponding standard_names are recommended instead.

Cheers

Jonathan

[rhorne@…]

Dear CF committee:

Three weeks has gone by without any additional comments on this enhancement.

Do you support this enhancement ?

very respectfully,

randy

[biard]

I'm good with it. (Not that it mattersmuch! :-)) I think the final solution sidesteps the thorny issues of data relationships that this ticket raised. It's also a good way to address the particular issues.

[jonathan]

Dear all

Assuming that Nan supports this proposal as it was most recently summarised ​in comment 52, then this ticket has sufficient support to be accepted according to the rules. Jeff, please could you include it in the new CF version you are currently compiling. Randy should be added to the list of additional authors of the CF standard in the new version. Thank you, Randy.

Jonathan

[ngalbraith]

I support this update - thanks, all.

[painter1]

Tanya Reshel has made the necessary changes to the CF Conventions document, for version 1.7. The two new standard_names have been approved. However, I am leaving this ticket open because the conformance document hasn't been updated. This also may affect any CF checker software, but I believe that the ticket can be closed without updating such software.