Efficiently Managing Large Scale Analytics Data in PI Vision

Handling large volumes of analytics data in PI Vision can be a complex task, especially when dealing with high variability in what's being measured, where, and when. A common scenario is monitoring multiple reactors, each with hundreds of potential substances that may be analyzed, as highlighted by a user who shared their experience of running HPLC analytics on approximately 100 reactors. The user faced a daunting prospect of creating up to 100,000 PI tags to accommodate all materials and their measurements, many of which may not be frequently used.

In such situations, creating 100,000 tags can indeed seem excessive and confusing, but there are strategies to manage this effectively.

Firstly, Understanding Your Data

1. Assessment of Measurement Frequency: Clearly understand which substances are measured frequently and which only occasionally. For frequently measured substances, creating PI tags might be justifiable, especially where historical trends are essential.

2. Database Consideration: Since the raw data is stored in a SQL database, leverage this existing infrastructure for infrequent data retrieval.

Strategies for Efficient Data Handling in PI Vision

1. Mixed Data Retrieval Methods:

   • PI AF Linked Tables: For substances that are rarely checked, consider linking the tables from your SQL database to PI AF. This method prevents the duplication of data storage in both SQL and PI Data Archive, though it limits the viewing to the current data point rather than historical trends.

2. Smart Tag Management:

   • Event Frames: Use Event Frames for tracking experiments, which can help manage the data without sprawling into hundreds of thousands of tags and still maintain flexibility in identifying projects based on substances used, without altering tag configurations consistently.

3. Custom Solutions

   • AF SDK and Custom Data References: Utilize AF SDK to create custom data references that allow for flexible querying. This can replicate some PI Point functionalities such as retrieving historical data, bridging the gap between SQL and PI Data Archive capabilities.

4. Optimized SQL Queries:

   • Write SQL queries to aggregate necessary data then import into PI Vision for real-time or historical trend analysis. This can potentially reduce the number of tags required by overlaying calculated data from SQL as needed.

Implementation and Flexibility

• Adaptability is Key: As the projects and reactors vary dynamically, ensure your system remains flexible. Event Frames or a centralized analytics system with versatile querying capabilities can inherently support this flexibility.

Keep in mind that while tags are a functional unit within the PI System designed for high-speed, low-latency data storage and retrieval, creative approaches to manage tag population can lead to resource reduction and increased clarity.