Data Sources and Prep FAQ

  • Not currently. We have kept the Flow Editor simple, and CSV is the simplest, most universal standard. We can support real-time feeds using JSON or other data types as part of a custom enterprise implementation, and we’d be happy to discuss that with you.

  • We have a limit of 60,000 points because performance on phones would degrade with millions of points. Flow is designed not for direct big data analysis, but for storytelling, so the complex analysis is best performed in Tableau, R-scripts, Python, Power BI, etc. We are not trying to compete in the data analysis space. There are a variety of mechanisms to summarize, random sample, or otherwise limit the total number of points. A sense of awe and wonder can be achieved with a few thousand points, and we love the drama of Flows that start with a large number of points that then drill into specifics.

  • Data is inserted into a Swarm using a CSV file.  A Flow can use any number of CSV files, and a given CSV file may be used to populate any number of Swarms.

    The first column in a CSV file contains the column names for the various columns of data being imported into the Swarm. Many of these column names will be shown as axis labels for Charts (and these cannot currently be changed within the Flow editor), so please make sure columns are labeled informatively and appropriately.

    It is good practice to make sure that the column names contain straightforward characters.  Characters such as tabs and newline characters will be accepted by Flow but using these types of characters may lead to maintenance issues later on.

    Values in a CSV file can be modified outside Flow and then re-imported into Flow, provided that the existing column names are not changed for columns that are referenced in filters, chart axes, etc.

    It is okay to add new columns into the new CSV file.

    It is risky to delete columns or change the spelling of existing column names.  Often columns are used to define axes in charts, provide screening logic in filters, and populate other settings within Flow.  Deleting or changing the spelling of columns that have been used for these functions will cause significant errors to occur within Flow.

    If you absolutely need to change column names that are referenced within Flow, please contact Flow Support for assistance.

    Note 1: You can easily upload a newly named CSV file to create a new Swarm, but depending on the circumstances, this may require you to revise the settings (for example, filter columns or axis labels) that you had set using a previous Swarm.

    Note 2: Future versions of Flow will include capabilities for processing several types of operations involving CSV files, and some of the issues described in this section will be handled internally by Flow in the future.

    A row in a CSV file that is to be imported into Flow may contain only one data point that is to be placed into the 3-D space for the data Swarm that is being populated by the CSV file.  If a row in an original CSV file contains multiple data points (for example, a time series row of data), the CSV file will need to be "unpivoted" so that only a single data point is contained in each row.  For example, in a CSV file containing yearly time-series information, an additional column named "Year" will be inserted into the unpivoted version of the CSV file.  Tableau, Excel, and other products have good ways of unpivoting CSV files, as unpivoting is required by various products that are used for data analysis.

  • Various Flow Editor settings can be specified within the CSV file for a Swarm.  Examples include dot sizes, dot colors, connection points, labels, text values, and others.
  • Sometimes the easiest way to provide complex record selection logic involving multiple data fields and data values is to use a new column in the CSV file to contain the results of the selection logic.  If you have done the complex calculations and recorded the logical results in a column within the CSV file, then relatively simple selection logic can be specified by referencing the newly constructed column within filters used in a Flow perspective.
  • If you are assigning locations based on a country, and the value of the country does not match the standard ISO value, the latitude and longitude of the country are defaulted to 0 and 0.  This combination of latitude and longitude lies off the Northwest coast of Africa.  The records that are placed at this location will be eliminated when all your country codes are set to a valid ISO value.
  • We now have a way to specify your own "N/A" value, and choose whether or not to have Flow ignore those values. Here are a list of the defaults: na, n/a, N/A, #N/A, NaN, NAN, N/a, Null, null, NULL, Ø, -, and #DIV/0!