From Newsgroup: comp.lang.tcl

Hi *,

I'd like a method to retrieve the complete key structure from a
json string and I'm using the json tcllib module.
It may contain an unknown number of nested levels and json arrays.
I have found some solutions which work basically I cannot
distinguish reliable between a normal - maybe nested json
object - and a json array.

All found examples fail json array arrays.

The extracted or created keys may eventually be used for accessing the corresponding values - json does not create specific ids for the array entries, they are a list in the tcl's point of view.

Has anyone tried or mastered this challenge?

Here's a typical example:
---
https://www.tech-edv.co.at/download/testdata/livedata_20250914.txt
--
Gerhard.Reithofer
http://www.tech-edv.co.at
-- new email address --
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

Gerhard Reithofer <[email protected]> wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a
json string and I'm using the json tcllib module.
It may contain an unknown number of nested levels and json arrays.
I have found some solutions which work basically I cannot
distinguish reliable between a normal - maybe nested json
object - and a json array.

This is because the tcllib json module does not return any typing
information, so you have to guess as to whether you have an "object" or
an "array" at any given level.

If you want to reliably parse unknown json and reliably know if you
have json objects or json arrays at any given level of the JSON tree,
your best bet is to parse the json with a newer Tdom that does preserve
such typing in the dom tree it outputs:

https://tdom.org/index.html/doc/trunk/doc/dom.html

-json
If -json is specified, the data is expected to be a valid JSON
string (according to RFC 7159). The command returns an ordinary
DOM document with nesting token inside the JSON data translated
into tree hierarchy. If a JSON array value is itself an object or
array then container element nodes named (in a default build)
arraycontainer or objectcontainer, respectively, are inserted into
the tree. The JSON serialization of this document (with the domDoc
method asJSON) is the same JSON information as the data, preserving
JSON datatypes, allowing non-unique member names of objects while
preserving their order and the full range of JSON string values.
JSON datatype handling is done with an additional property
"sticking" at the doc and tree nodes. This property isn't
contained in an XML serialization of the document. If you need to
store the JSON data represented by a document, store the JSON
serialization and parse it back from there. Apart from this JSON
type information the returned doc command or handle is an ordinary
DOM doc, which may be investigated or modified with the full range
of the doc and node methods. Please note that the element node
names and the text node values within the tree may be outside of
what the appropriate XML productions allow.

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a
json string and I'm using the json tcllib module.
It may contain an unknown number of nested levels and json arrays.
I have found some solutions which work basically I cannot
distinguish reliable between a normal - maybe nested json
object - and a json array.

All found examples fail json array arrays.

The extracted or created keys may eventually be used for accessing the corresponding values - json does not create specific ids for the array entries, they are a list in the tcl's point of view.

Has anyone tried or mastered this challenge?

Here's a typical example:
---
https://www.tech-edv.co.at/download/testdata/livedata_20250914.txt

I'm by no means a JSON expert, in natural language terms I'd describe
myself as speaking schoolboy JSON, so I may have overlooked some
technical subtlety. As far as I can see

package require json
package require http
package require tls

http::register https 443 {tls::socket -autoservername true}

set tok [http::geturl https://www.tech-edu.co.at/download/testdata/livedata_20290914.txt]
set jsStr [http::data $tok]
http::cleanup $tok

set jsDict [json::json2dict $jsStr]

delivers a completely usable dictionary.

dict get $jsDict inverters

returns a two-element list which is analogous, in many programming
languages, to an array with two valid indexes.

Sorry if I've missed something.


Alan
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

In article <10abki9$2ift8$[email protected]>,
Alan Grunwald <[email protected]> wrote:

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a
json string and I'm using the json tcllib module.
It may contain an unknown number of nested levels and json arrays.
I have found some solutions which work basically I cannot
distinguish reliable between a normal - maybe nested json
object - and a json array.

All found examples fail json array arrays.

The extracted or created keys may eventually be used for accessing the
corresponding values - json does not create specific ids for the array
entries, they are a list in the tcl's point of view.

Has anyone tried or mastered this challenge?

Here's a typical example:
---
https://www.tech-edv.co.at/download/testdata/livedata_20250914.txt

I'm by no means a JSON expert, in natural language terms I'd describe
myself as speaking schoolboy JSON, so I may have overlooked some
technical subtlety. As far as I can see

package require json
package require http
package require tls

http::register https 443 {tls::socket -autoservername true}

set tok [http::geturl
https://www.tech-edu.co.at/download/testdata/livedata_20290914.txt]
set jsStr [http::data $tok]
http::cleanup $tok

set jsDict [json::json2dict $jsStr]

delivers a completely usable dictionary.

dict get $jsDict inverters

returns a two-element list which is analogous, in many programming >languages, to an array with two valid indexes.

Sorry if I've missed something.

Alan

I'm pretty sure you could do this fairly easily with the rl_json package.
The disadvantage being that it's not a standard part of tcllib.

https://github.com/RubyLane/rl_json
--
columbiaclosings.com
What's not in Columbia anymore..
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Mon, 15 Sep 2025, Rich wrote:

Gerhard Reithofer <[email protected]> wrote:

Hi *,

...

This is because the tcllib json module does not return any typing information, so you have to guess as to whether you have an "object" or
an "array" at any given level.

...

Thanks, but this is rather powerful but also heavy set for that "simple" problem.

But if I find no simple solution in short time I will come back to it
:-)

THX
--
mailto:[email protected]
http://www.tech-edv.co.at
-- new email address --
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Tue, 16 Sep 2025, Alan Grunwald wrote:
Hi Alan,

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a
json string and I'm using the json tcllib module.
It may contain an unknown number of nested levels and json arrays.

...

delivers a completely usable dictionary.

dict get $jsDict inverters

returns a two-element list which is analogous, in many programming languages, to an array with two valid indexes.

yes, it is a two-element list but none to use it as tcl array or dict:

set inv1 "[string range [dict get $jsDict inverters] 0 86] ..."
{serial 123456789012 name {HM-400 Extra} order 0 data_age 4 poll_enabled true reachable ...

set inv2 "[string range [dict get $jsDict inverters] 1 86] ..."
serial 210987654321 name {HM-400 Extra} order 0 data_age 4 poll_enabled true reachable ...

There is no 1st element which can be used as index, it is a plain list.
That this item has only 2 enries is by chance, there can be also a
single record but also 100th of them.

On the other side a typical json-objects are:
dict get $jsDict total =>
Power {v 207 u W d 1} YieldDay {v 1356 u Wh d 0} YieldTotal {v 1584.98 u kWh d 3}

These are 3 correct json-objects with the keys Power, YieldDay and YieldTotal. Each of these objects represents a numric (v)alue, (u)nit and a number
of (d)ecimal and places ... i think ;-(

The 2 situations - json array (in tcl a list) and json object (in tcl
dict or array with key value) can be mixed.

Still my main problem is to dinguish these 2 cases!

Thank you very much,
Gerhard
--
mailto:[email protected]
http://www.tech-edv.co.at
-- new email address --
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Tue, 16 Sep 2025, Ted Nolan <tednolan> wrote:
Hi Ted,

In article <10abki9$2ift8$[email protected]>,
Alan Grunwald <[email protected]> wrote:

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a

...

The disadvantage being that it's not a standard part of tcllib.

https://github.com/RubyLane/rl_json

unfortunately I haven't found the complate documentation online - any
hint?

BTW I think that this problem can be solved with various tools, I have
only tried it with tcllib json and I'm a fan of tcl-only
implementations because tcl is availale on many platform and then you
need not to make anything except installing of tcl and if necessary
copy a bunch of files.

Thank you,
Gerhard
--
mailto:[email protected]
http://www.tech-edv.co.at
-- new email address --
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 9/16/2025 3:50 PM, Gerhard Reithofer wrote:

...

The disadvantage being that it's not a standard part of tcllib.

https://github.com/RubyLane/rl_json

unfortunately I haven't found the complate documentation online - any
hint?

I haven't used it but the link posted above seems to have a pretty good description. If you are after a how-to-use kind of manual, which would
be nice to have, very few packages have it and they are usually found in
a book.

By the way, the url to the sample file gives an error.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

In article <[email protected]>,
Gerhard Reithofer <[email protected]> wrote:

On Tue, 16 Sep 2025, Ted Nolan <tednolan> wrote:
Hi Ted,

In article <10abki9$2ift8$[email protected]>,
Alan Grunwald <[email protected]> wrote:

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a

...

The disadvantage being that it's not a standard part of tcllib.

https://github.com/RubyLane/rl_json

unfortunately I haven't found the complate documentation online - any
hint?

The man page is at the bottom of that link, below the file listing.
--
columbiaclosings.com
What's not in Columbia anymore..
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

In article <[email protected]>,
Gerhard Reithofer <[email protected]> wrote:

On Tue, 16 Sep 2025, Ted Nolan <tednolan> wrote:
Hi Ted,

In article <10abki9$2ift8$[email protected]>,
Alan Grunwald <[email protected]> wrote:

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a

...

The disadvantage being that it's not a standard part of tcllib.

https://github.com/RubyLane/rl_json

unfortunately I haven't found the complate documentation online - any
hint?

BTW I think that this problem can be solved with various tools, I have
only tried it with tcllib json and I'm a fan of tcl-only
implementations because tcl is availale on many platform and then you
need not to make anything except installing of tcl and if necessary
copy a bunch of files.

Thank you,
Gerhard

To follow up, assuming you have already downloaded the
data, something like the below is possible, though
no decent output formatting is done in this little script:


===================CUT HERE==================
#!/usr/bin/env tclsh8.6

lappend auto_path /usr/local/cluster/rl_json_0.9.12/lib

package require rl_json
namespace import ::rl_json::*

proc print_array {ar} {

set i 0

json foreach val $ar {
puts "Element $i"
if {[json type $val] in {"string" "number" "boolean" "null"}} {
puts " [json get $val]"
} elseif {[json type $val] == "object"} {
print_object $val
} elseif {[json type $val] == "array"} {
print_array $val
} else {
puts stderr "Ooops should not happen!"
exit 1
}
incr i
}
}

proc print_object {obj} {

json foreach {key val} $obj {
puts $key
if {[json type $val] in {"string" "number" "boolean" "null"}} {
puts " [json get $val]"
} elseif {[json type $val] == "object"} {
print_object $val
} elseif {[json type $val] == "array"} {
print_array $val
} else {
puts stderr "Ooops should not happen!"
exit 1
}
}
}

proc main {} {

set f [open {livedata_20250914.txt} r]
fconfigure $f -encoding utf-8

set data [read $f]
close $f

# puts $data

set toplevel [json type $data]

# A toplevel can only be an "object" or an "array"
if {$toplevel == "object"} {
print_object $data
} else {
print_array $data
}

}

main
================END=============
--
columbiaclosings.com
What's not in Columbia anymore..
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 9/16/2025 12:50 PM, Gerhard Reithofer wrote:

On Tue, 16 Sep 2025, Ted Nolan <tednolan> wrote:
Hi Ted,

In article <10abki9$2ift8$[email protected]>,
Alan Grunwald <[email protected]> wrote:

On 15/09/2025 17:58, Gerhard Reithofer wrote:

Hi *,

I'd like a method to retrieve the complete key structure from a

...

The disadvantage being that it's not a standard part of tcllib.

https://github.com/RubyLane/rl_json

unfortunately I haven't found the complate documentation online - any
hint?

BTW I think that this problem can be solved with various tools, I have
only tried it with tcllib json and I'm a fan of tcl-only
implementations because tcl is availale on many platform and then you
need not to make anything except installing of tcl and if necessary
copy a bunch of files.

Thank you,
Gerhard

I posed this problem to the claude AI, and it agrees with your findings of loss of type info. Claude's suggestion was to preparse the json string to find array keys, and supplied the following code using regex's:

package require json

proc findArrayKeys {jsonString {path {}}} {
set arrayKeys {}

# Remove whitespace for easier parsing
set json [string map {"\n" "" "\t" "" " " " "} $jsonString]

# Find "key": [ patterns (arrays)
set pattern {"([^"]+)"\s*:\s*\[}
set start 0
while {[regexp -start $start -indices $pattern $json match keyIndices]} {
set key [string range $json {*}$keyIndices]
if {$path ne ""} {
lappend arrayKeys "$path.$key"
} else {
lappend arrayKeys $key
}
set start [expr {[lindex $match 1] + 1}]
}

return $arrayKeys
}

# Example usage
set jsonData {{"items": ["a", "b", "c"], "single": "hello", "nested": {"subitems": ["x", "y"]}}}
set parsed [::json::json2dict $jsonData]
set arrayKeys [findArrayKeys $jsonData]

puts "Array keys: $arrayKeys"

foreach {key value} $parsed {
if {$key in $arrayKeys} {
puts "$key is an array: $value"
} else {
puts "$key is not an array: $value"
}
}
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On Tue, 16 Sep 2025, et99 wrote:

On 9/16/2025 12:50 PM, Gerhard Reithofer wrote:

On Tue, 16 Sep 2025, Ted Nolan <tednolan> wrote:
Hi Ted,

...

I posed this problem to the claude AI, and it agrees with your findings of loss of type info. Claude's suggestion was to preparse the json string to find
array keys, and supplied the following code using regex's:

package require json

proc findArrayKeys {jsonString {path {}}} {
set arrayKeys {}
# Remove whitespace for easier parsing
set json [string map {"\n" "" "\t" "" " " " "} $jsonString]
# Find "key": [ patterns (arrays)
set pattern {"([^"]+)"\s*:\s*\[}
set start 0
while {[regexp -start $start -indices $pattern $json match keyIndices]} {
set key [string range $json {*}$keyIndices]
if {$path ne ""} {
lappend arrayKeys "$path.$key"
} else {
lappend arrayKeys $key
}
set start [expr {[lindex $match 1] + 1}]
}
return $arrayKeys
}

# Example usage
set jsonData {{"items": ["a", "b", "c"], "single": "hello", "nested": {"subitems": ["x", "y"]}}}
set parsed [::json::json2dict $jsonData]
set arrayKeys [findArrayKeys $jsonData]

puts "Array keys: $arrayKeys"

foreach {key value} $parsed {
if {$key in $arrayKeys} {
puts "$key is an array: $value"
} else {
puts "$key is not an array: $value"
}
}

Really interesting approach good looks good.
Not mentioning that all entities must be reparsed recursively it could
be a solution.

The results for my examples "A" and "O" refer to array and object:
Example: livedata_20250413.json => Array keys: inverters
inverters A {serial 429412742452 name {HM-800 am Speicher} order 0 data_age 17 poll_enabled true re ...
total O Power {v 201.6999969 u W d 1} YieldDay {v 1776 u Wh d 0} YieldTotal {v 1015.158997 u kWh d 3}
hints O time_sync false radio_problem false default_password false

Example: livedata_20250914.json => Array keys: inverters
inverters A {serial 112183843984 name {HM-400 Extra} order 0 data_age 4 poll_enabled true reachable ...
total O Power {v 207 u W d 1} YieldDay {v 1356 u Wh d 0} YieldTotal {v 1584.98 u kWh d 3}
hints O time_sync false radio_problem false default_password false vedirect O enabled false
huawei O enabled false
battery O enabled false
power_meter O enabled true Power {v 54 u W d 1}

:-)
--
mailto:[email protected]
http://www.tech-edv.co.at
-- new email address --
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: comp.lang.tcl

On 9/17/2025 5:53 AM, Gerhard Reithofer wrote:

On Tue, 16 Sep 2025, et99 wrote:

On 9/16/2025 12:50 PM, Gerhard Reithofer wrote:

On Tue, 16 Sep 2025, Ted Nolan <tednolan> wrote:
Hi Ted,

...

I posed this problem to the claude AI, and it agrees with your findings of >> loss of type info. Claude's suggestion was to preparse the json string to find
array keys, and supplied the following code using regex's:

package require json

proc findArrayKeys {jsonString {path {}}} {
set arrayKeys {}
# Remove whitespace for easier parsing
set json [string map {"\n" "" "\t" "" " " " "} $jsonString]
# Find "key": [ patterns (arrays)
set pattern {"([^"]+)"\s*:\s*\[}
set start 0
while {[regexp -start $start -indices $pattern $json match keyIndices]} {
set key [string range $json {*}$keyIndices]
if {$path ne ""} {
lappend arrayKeys "$path.$key"
} else {
lappend arrayKeys $key
}
set start [expr {[lindex $match 1] + 1}]
}
return $arrayKeys
}

# Example usage
set jsonData {{"items": ["a", "b", "c"], "single": "hello", "nested":
{"subitems": ["x", "y"]}}}
set parsed [::json::json2dict $jsonData]
set arrayKeys [findArrayKeys $jsonData]

puts "Array keys: $arrayKeys"

foreach {key value} $parsed {
if {$key in $arrayKeys} {
puts "$key is an array: $value"
} else {
puts "$key is not an array: $value"
}
}

Really interesting approach good looks good.
Not mentioning that all entities must be reparsed recursively it could
be a solution.

I agree, it looks good. I am glad it will work for you.

--- Synchronet 3.21a-Linux NewsLink 1.2