NAS: Storing data for use between scripts (pre/post processing)

Discussion created by carstein.seeberg on Jun 28, 2010

A feature of the NAS scripting that deserves some exposure is the setvariable and getvariable functions.

These functions take a key/value pair (both strings) that end up in a global hash structure that lives as long as the NAS runs.  A typical use-case could be to use the scheduler to load the variable hash with data from e.g. a user-database using the database class, or a file or nimbus requests.  The scheduler has an option to run-once (mode: By Time).


The value data is required to be a string, however this example will also show how to use a complex datatype (table) as storage for your data structure.  The hint is to serialize the table.


The code to serialize the table is found here , I added the code (and modified it alittle)  to a library that I called table-lib (complete code is below) .


I prepared a table called my_cache in the user database (user.db)

    CREATE TABLE my_cache (name text,value text);

and added stuff to it:

     INSERT INTO my_cache(name,value) VALUES ('***','123');
     INSERT INTO my_cache(name,value) VALUES ('yyy','234');


I created a scheduled script (run-once) to load the cache:


require ("library/table-lib")  -- Load cache printf ("CACHE Loaded....") rs = database.query("SELECT * FROM my_cache") for i=1,#rs do    setvariable(trim(rs[i].name),[i]) ) end database.close() setvariable("mycache.loaded", tostring( 


Or you can add data from a different script:



require ("library/table-lib")  local t = {}  = "my-name" t.value = "my-value" t.func  = function(x,y)                print( "Hello World" )                local sum = x+y                return sum           end  setvariable ("mytable",



You can then access this data from a different script (pre-processing script):


require ("library/table-lib")  local lookup = getvariable(event.hostname)  if lookup ~= nil then    local t = table.load ( lookup )    printf ("CACHE: '%s' found, name: %s, value: %s",event.hostname,, t.value)    event.custom_1 = "CACHED: ".. .. " VALUE:" .. t.value else    printf ("CACHE: '%s' NOT found",event.hostname) end  return event



Or a in a post-processing script:


require ("library/table-lib")  local t2 = table.load (getvariable ("mytable"))  -- note the use of a function in the complex datatype - pretty cool:smileyhappy:printf (" %s, sum: %d",, t2.func(1,2) )



As you can see, a pretty powerful mechanism!  The table extensions -,table.load() and table.print() are all found in the library/table-lib file.  The and table.load also supports serializing to file.


Keep on scripting!




Add the following code to library/table-lib:


-- -- extensions to the table library -- --[[    Save Table to File/Stringtable    Load Table from File/Stringtable    v 0.94        Lua 5.1 compatible        Userdata and indices of these are not saved    Functions are saved via string.dump, so make sure it has no upvalues    References are saved    ---------------------------------------------------- table [, filename] )        Saves a table so it can be called via the table.load function again    table must a object of type 'table'    filename is optional, and may be a string representing a filename or true/1 table )       on success: returns a string representing the table (stringtable)       (uses a string as buffer, ideal for smaller tables) table, true or 1 )       on success: returns a string representing the table (stringtable)       (uses io.tmpfile() as buffer, ideal for bigger tables) table, "filename" )       on success: returns 1       (saves the table to file "filename")    on failure: returns as second argument an error msg    ----------------------------------------------------    table.load( filename or stringtable )        Loads a table that has been saved via the function        on success: returns a previously saved table    on failure: returns as second argument an error msg    ----------------------------------------------------        chillcode,   Licensed under the same terms as Lua itself.      Carstein added table.print()  --]]  -- declare local variables --// exportstring( string ) --// returns a "Lua" portable version of the string local function exportstring( s )    s = string.format( "%q",s )    -- to replace    s = string.gsub( s,"\\\n","\\n" )    s = string.gsub( s,"\r","\\r" )    s = string.gsub( s,string.char(26),"\"..string.char(26)..\"" )    return s end  --// The Save Function function  tbl,filename )    local charS,charE = "   ","\n"    local file,err    -- create a pseudo file that writes to a string and return the string    if not filename then       file =  { write = function( self,newstr ) self.str = self.str..newstr end, str = "" }       charS,charE = "",""    -- write table to tmpfile    elseif filename == true or filename == 1 then       charS,charE,file = "","",io.tmpfile()    -- write table to file    -- use here rather than io.output, since in windows when clicking on a file opened with io.output will create an error    else       file,err = filename, "w" )       if err then return _,err end    end    -- initiate variables for save procedure    local tables,lookup = { tbl },{ [tbl] = 1 }    file:write( "return {"..charE )    for idx,t in ipairs( tables ) do       if filename and filename ~= true and filename ~= 1 then          file:write( "-- Table: {"..idx.."}"..charE )       end       file:write( "{"..charE )       local thandled = {}       for i,v in ipairs( t ) do          thandled[i] = true          -- escape functions and userdata          if type( v ) ~= "userdata" then             -- only handle value             if type( v ) == "table" then                if not lookup[v] then                   table.insert( tables, v )                   lookup[v] = #tables                end                file:write( charS.."{"..lookup[v].."},"..charE )             elseif type( v ) == "function" then                file:write( charS.."loadstring("..exportstring(string.dump( v )).."),"..charE )             else                local value =  ( type( v ) == "string" and exportstring( v ) ) or tostring( v )                file:write(  charS..value..","..charE )             end          end       end       for i,v in pairs( t ) do          -- escape functions and userdata          if (not thandled[i]) and type( v ) ~= "userdata" then             -- handle index             if type( i ) == "table" then                if not lookup[i] then                   table.insert( tables,i )                   lookup[i] = #tables                end                file:write( charS.."[{"..lookup[i].."}]=" )             else                local index = ( type( i ) == "string" and "["..exportstring( i ).."]" ) or string.format( "[%d]",i )                file:write( charS..index.."=" )             end             -- handle value             if type( v ) == "table" then                if not lookup[v] then                   table.insert( tables,v )                   lookup[v] = #tables                end                file:write( "{"..lookup[v].."},"..charE )             elseif type( v ) == "function" then                file:write( "loadstring("..exportstring(string.dump( v )).."),"..charE )             else                local value =  ( type( v ) == "string" and exportstring( v ) ) or tostring( v )                file:write( value..","..charE )             end          end       end       file:write( "},"..charE )    end    file:write( "}" )    -- Return Values    -- return stringtable from string    if not filename then       -- set marker for stringtable       return file.str.."--|"    -- return stringttable from file    elseif filename == true or filename == 1 then       file:seek ( "set" )       -- no need to close file, it gets closed and removed automatically       -- set marker for stringtable       return file:read( "*a" ).."--|"    -- close file and return 1    else       file:close()       return 1    end end  --// The Load Function function table.load( sfile )    -- catch marker for stringtable    if string.sub( sfile,-3,-1 ) == "--|" then       tables,err = loadstring( sfile )    else       tables,err = loadfile( sfile )    end    if err then return _,err    end    tables = tables()    for idx = 1,#tables do       local tolinkv,tolinki = {},{}       for i,v in pairs( tables[idx] ) do          if type( v ) == "table" and tables[v[1]] then             table.insert( tolinkv,{ i,tables[v[1]] } )          end          if type( i ) == "table" and tables[i[1]] then             table.insert( tolinki,{ i,tables[i[1]] } )          end       end       -- link values, first due to possible changes of indices       for _,v in ipairs( tolinkv ) do          tables[idx][v[1]] = v[2]       end       -- link indices       for _,v in ipairs( tolinki ) do          tables[idx][v[2]],tables[idx][v[1]] =  tables[idx][v[1]],nil       end    end    return tables[1] end  function table.print ( t,tab,lookup )      local lookup = lookup or { [t] = 1 }      local tab = tab or ""      for i,v in pairs( t ) do           print( tab..tostring(i), v )           if type(i) == "table" and not lookup[i] then                lookup[i] = 1                print( tab.."Table: i" )                table.print (i,tab.."\t",lookup )           end           if type(v) == "table" and not lookup[v] then                lookup[v] = 1                print( tab.."Table: v" )                table.print (v,tab.."\t",lookup )           end      end end