Static and runtime analysis of computer program systems
28 claims: 5 independent, 23 dependent
- 1複数のコンピュータプログラムを含むコンピュータプログラムエコシステムを分析するためのコンピュータによって実装される方法であって、前記コンピュータプログラムエコシステムに関連する1つ又は複数のパラメータセット内のパラメータの値に基づいて前記コンピュータプログラムエコシステムの要素間の静的依存関係を識別するステップであって、前記コンピュータプログラムエコシステムの前記要素は前記コンピュータプログラムエコシステムの前記複数のコンピュータプログラム及び前記コンピュータプログラムに関連するデータ資源を含む、識別するステップを含む、前記コンピュータプログラムエコシステムの静的分析を行うステップと、データレコードを処理するために前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記要素の1つ又は複数を識別するステップを含む、前記コンピュータプログラムエコシステムの実行時分析を行うステップと、前記コンピュータプログラムエコシステムの別のコンピュータプログラムへのスケジュール依存関係を有する前記コンピュータプログラムエコシステムのコンピュータプログラムを識別するステップを含む、前記コンピュータプログラムエコシステムのスケジュール分析を行うステップと、前記静的分析、前記実行時分析、及び前記スケジュール分析の結果に基づき、前記コンピュータプログラムエコシステムの前記要素のサブセットをエコシステムユニットとして識別するステップと、第1のコンピュータシステムから第2のコンピュータシステムに前記エコシステムユニットを移行するか、前記エコシステムユニットを試験するか、又はその両方を行うステップとを含む方法。
- 2実行時分析を行うステップが、前記コンピュータプログラムエコシステムの実行によって生成される実行時ログ情報を得るステップと、前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記1つ又は複数の要素を識別するために前記実行時ログ情報を分析するステップとを含む、請求項1に記載の方法。
- 3前記実行時ログ情報を分析するステップが、前記コンピュータプログラムエコシステムの実行中に実行された1つ又は複数のコンピュータプログラムを識別するために前記実行時ログ情報内のパラメータの値を分析するステップを含む、請求項2に記載の方法。
- 4前記要素のサブセットをエコシステムユニットとして識別するステップが、前記コンピュータプログラムエコシステムの実行中に実行された前記コンピュータプログラムを前記エコシステムユニットの一部として識別するステップを含む、請求項3に記載の方法。
- 5前記コンピュータプログラムエコシステムの実行中に実行された1つ又は複数のコンピュータプログラムを識別するステップが、実行されたコンピュータプログラム間の実行時依存関係を識別するステップを含む、請求項3に記載の方法。
- 6実行されたコンピュータプログラム間の実行時依存関係を識別するステップが、前記実行時ログ情報内の前記パラメータの前記値に基づいて前記実行時依存関係を識別するステップを含む、請求項3に記載の方法。
- 7前記実行時ログ情報内のパラメータの値を分析するステップが、前記値の1つ又は複数を一般化するステップを含む、請求項3に記載の方法。
- 8前記実行時ログ情報を分析するステップが、前記コンピュータプログラムエコシステムの実行中にアクセスされた1つ又は複数のデータ資源を識別するために前記実行時ログ情報内のパラメータの値を分析するステップを含む、請求項3に記載の方法。
- 9前記コンピュータプログラムエコシステムの実行中にアクセスされた各データ資源の名前を明らかにするために前記実行時ログ情報内の前記パラメータの前記値を分析するステップをさらに含む、請求項8に記載の方法。
- 10前記要素のサブセットをエコシステムユニットとして識別するステップが、前記コンピュータプログラムエコシステムの実行中にアクセスされた前記データ資源を前記エコシステムユニットの一部として識別するステップを含む、請求項8に記載の方法。
- 11前記コンピュータプログラムエコシステムの前記要素のサブセットをエコシステムユニットとして識別するステップが、前記コンピュータプログラムエコシステムの実行中に利用された前記1つ又は複数の要素を前記エコシステムユニット内に含めるステップを含む、請求項1に記載の方法。
- 12スケジュール分析を行うステップがスケジューラからスケジュール情報を得るステップを含み、前記スケジューラは前記コンピュータプログラムエコシステムと別個である、請求項1に記載の方法。
- 13スケジュール分析を行うステップが、前記コンピュータプログラムエコシステムの第2のコンピュータプログラムの実行後に実行される予定の前記コンピュータプログラムエコシステムの第1のコンピュータプログラムを識別するステップを含む、請求項1に記載の方法。
- 14前記要素のサブセットをエコシステムユニットとして識別するステップが、前記第1のコンピュータプログラム及び前記第2のコンピュータプログラムを前記エコシステムユニットの一部として識別するステップを含む、請求項13に記載の方法。
- 15データを処理するために前記コンピュータプログラムエコシステムを実行するステップをさらに含む、請求項1に記載の方法。
- 16前記コンピュータプログラムエコシステムを実行するステップが、前記コンピュータプログラムエコシステムに関連する前記パラメータセットの1つ又は複数の中のパラメータの値に従って前記コンピュータプログラムエコシステムの前記コンピュータプログラムの1つ又は複数をインスタンス化するステップを含む、請求項15に記載の方法。
- 17前記コンピュータプログラムエコシステムを実行するステップが、前記データを処理するために前記1つ又は複数のインスタンス化されたコンピュータプログラムを実行するステップを含む、請求項16に記載の方法。
- 18実行時分析を行うステップが、 前記コンピュータプログラムエコシステムの実行によって生成される実行時ログ情報を得るステップと、 前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記1つ又は複数の要素を識別するために前記実行時ログ情報を分析するステップと を含み、 前記実行時ログ情報が、1つ又は複数の実行されるインスタンス化されたコンピュータプログラムを示す情報を含む、請求項17に記載の方法。
- 19前記コンピュータプログラムエコシステムを実行するステップが、前記コンピュータプログラムエコシステムに関連する前記データ資源の1つ又は複数にアクセスするステップを含む、請求項15に記載の方法。
- 20実行時分析を行うステップが、 前記コンピュータプログラムエコシステムの実行によって生成される実行時ログ情報を得るステップと、 前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記1つ又は複数の要素を識別するために前記実行時ログ情報を分析するステップと を含み、 前記実行時ログ情報が、1つ又は複数のアクセスされるデータ資源を示す情報を含む、請求項19に記載の方法。
- 21前記コンピュータプログラムエコシステムに関連する前記データ資源の1つ又は複数にアクセスするステップが、ファイル、データベース、及びオブジェクトの1つ又は複数からデータを読み出すステップを含む、請求項19に記載の方法。
- 22前記コンピュータプログラムエコシステムに関連する前記データ資源の1つ又は複数にアクセスするステップが、ファイル、データベース、及びオブジェクトの1つ又は複数にデータを書き込むステップを含む、請求項19に記載の方法。
- 23前記コンピュータプログラムエコシステムを実行するステップが、 実 行時ログ情報を生成するステップを含む、請求項19に記載の方法。
- 24実 行時ログ情報を生成するステップが、前記実行時ログ情報をログファイルに書き込むステップを含む、請求項23に記載の方法。
- 25前記1つ又は複数のコンピュータプログラムがデータフローグラフを含む、請求項1に記載の方法。
- 26複数のコンピュータプログラムを含むコンピュータプログラムエコシステムを分析するためのコンピュータシステムであって、メモリに接続された1つ又は複数のプロセッサを含み、前記1つ又は複数のプロセッサが、前記コンピュータプログラムエコシステムに関連する1つ又は複数のパラメータセット内のパラメータの値に基づいて前記コンピュータプログラムエコシステムの要素間の静的依存関係を識別することであって、前記コンピュータプログラムエコシステムの前記要素は前記コンピュータプログラムエコシステムの前記複数のコンピュータプログラム及び前記コンピュータプログラムに関連するデータ資源を含む、識別することを含む、前記コンピュータプログラムエコシステムの静的分析を行うことと、データレコードを処理するために前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記要素の1つ又は複数を識別することを含む、前記コンピュータプログラムエコシステムの実行時分析を行うことと、前記コンピュータプログラムエコシステムの別のコンピュータプログラムへのスケジュール依存関係を有する前記コンピュータプログラムエコシステムのコンピュータプログラムを識別することを含む、前記コンピュータプログラムエコシステムのスケジュール分析を行うことと、前記静的分析、前記実行時分析、及び前記スケジュール分析の結果に基づき、前記コンピュータプログラムエコシステムの前記要素のサブセットをエコシステムユニットとして識別することと、第1のコンピュータシステムから第2のコンピュータシステムに前記エコシステムユニットを移行するか、前記エコシステムユニットを試験するか、又はその両方を行うこととを行うように構成される、コンピュータシステム。
- 27複数のコンピュータプログラムを含むコンピュータプログラムエコシステムをコンピュータシステムに分析させるための命令を記憶した、非一時的なコンピュータ可読媒体であって、前記命令が、前記コンピュータシステムに、前記コンピュータプログラムエコシステムに関連する1つ又は複数のパラメータセット内のパラメータの値に基づいて前記コンピュータプログラムエコシステムの要素間の静的依存関係を識別することであって、前記コンピュータプログラムエコシステムの前記要素は前記コンピュータプログラムエコシステムの前記複数のコンピュータプログラム及び前記コンピュータプログラムに関連するデータ資源を含む、識別することを含む、前記コンピュータプログラムエコシステムの静的分析を行うことと、データレコードを処理するために前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記要素の1つ又は複数を識別することを含む、前記コンピュータプログラムエコシステムの実行時分析を行うことと、前記コンピュータプログラムエコシステムの別のコンピュータプログラムへのスケジュール依存関係を有する前記コンピュータプログラムエコシステムのコンピュータプログラムを識別することを含む、前記コンピュータプログラムエコシステムのスケジュール分析を行うことと、前記静的分析、前記実行時分析、及び前記スケジュール分析の結果に基づき、前記コンピュータプログラムエコシステムの前記要素のサブセットをエコシステムユニットとして識別することと、第1のコンピュータシステムから第2のコンピュータシステムに前記エコシステムユニットを移行するか、前記エコシステムユニットを試験するか、又はその両方を行うこととを行わせる、非一時的なコンピュータ可読媒体。
- 28複数のコンピュータプログラムを含むコンピュータプログラムエコシステムを分析するためのコンピュータシステムであって、前記コンピュータプログラムエコシステムに関連する1つ又は複数のパラメータセット内のパラメータの値に基づいて前記コンピュータプログラムエコシステムの要素間の静的依存関係を識別することであって、前記コンピュータプログラムエコシステムの前記要素は前記コンピュータプログラムエコシステムの前記複数のコンピュータプログラム及び前記コンピュータプログラムに関連するデータ資源を含む、識別することを含む、前記コンピュータプログラムエコシステムの静的分析を行うための手段と、データレコードを処理するために前記コンピュータプログラムエコシステムの実行中に利用された前記コンピュータプログラムエコシステムの前記要素の1つ又は複数を識別することを含む、前記コンピュータプログラムエコシステムの実行時分析を行うための手段と、前記コンピュータプログラムエコシステムの別のコンピュータプログラムへのスケジュール依存関係を有する前記コンピュータプログラムエコシステムのコンピュータプログラムを識別することを含む、前記コンピュータプログラムエコシステムのスケジュール分析を行うための手段と、前記静的分析、前記実行時分析、及び前記スケジュール分析の結果に基づき、前記コンピュータプログラムエコシステムの前記要素のサブセットをエコシステムユニットとして識別するための手段と、第1のコンピュータシステムから第2のコンピュータシステムに前記エコシステムユニットを移行するか、前記エコシステムユニットを試験するか、又はその両方を行うための手段とを含むコンピュータシステム。
Independent claims28
116 paragraphs, as filed
Claiming Priority This application claims priority under 35 U.S.C. 119(e) to U.S. patent application Ser. No. 16/746,392, filed on May 1, 2003, the entire contents of both of which are hereby incorporated by reference.
A data processing system may include a plurality of computer programs executable to process input data. Within a data processing system, data may be passed from one computer program to another computer program to generate a set of output data.
<p><patcit num="1"><text>U.S. Patent Application Publication No. 2016/0019057</text></patcit><patcit num="2"><text>U.S. Patent Application Publication No. 2007/0011668</text></patcit><patcit num="3"><text>U.S. Pat. No. 5,966,072</text></patcit></p>
<p>The inventors analyze computer programs and associated data resources (referred to herein as computer program ecosystems) to identify subsets of computer programs and data resources as ecosystem units, and Techniques for performing one or more system tasks using an ecosystem unit are described herein. Computer programs and data resources within an ecosystem unit can be identified based on their static and/or runtime dependencies on each other. When the computer program ecosystem is executed, particular ones of the computer programs are executed and particular data resources are accessed by the executing ones of the computer programs. Ecosystem units can be identified by comparing static dependencies with information indicating which computer programs were running and which data resources were accessed. Treating computer programs and data resources that form an ecosystem unit as a single entity for performing system tasks such as migrating from one computing system to another or testing a computer program ecosystem is a system task. can promote Treating ecosystem units collectively for system tasks allows all relevant elements of the computer program ecosystem to be included in the execution of system tasks, while, for example, thereby consuming less processing power for system tasks. The system's computational data resources (eg, memory, processing power, processing time) can be conserved while allowing for more efficient execution.</p><p>In one aspect, a computer-implemented method for analyzing a computer program ecosystem including a plurality of computer programs comprises: analyzing a computer program based on values of parameters in one or more parameter sets associated with the computer program ecosystem; identifying static dependencies between elements of a program ecosystem, the elements of the computer program ecosystem including a plurality of computer programs of the computer program ecosystem and data resources associated with the computer programs; performing a static analysis of the computer program ecosystem. The method performs a run-time analysis of the computer program ecosystem including identifying one or more of the elements of the computer program ecosystem utilized during execution of the computer program ecosystem to process data records. Including steps. The method includes performing a schedule analysis of the computer program ecosystem including identifying computer programs of the computer program ecosystem that have a schedule dependency on another computer program of the computer program ecosystem. The method includes identifying a subset of elements of the computer program ecosystem as ecosystem units based on the results of static analysis, runtime analysis, and schedule analysis. The method includes migrating the ecosystem unit from the first computer system to the second computer system, testing the ecosystem unit, or both.</p><p>Embodiments can include any combination of one or more of the following features.</p><p>Performing run-time analysis includes obtaining run-time log information generated by execution of the computer program ecosystem and analyzing one or more elements of the computer program ecosystem utilized during execution of the computer program ecosystem. and analyzing the runtime log information to identify. Analyzing the runtime log information includes analyzing values of parameters within the runtime log information to identify one or more computer programs executed during execution of the computer program ecosystem. Identifying the subset of elements as an ecosystem unit includes identifying computer programs executed during execution of the computer program ecosystem as part of the ecosystem unit. Identifying one or more computer programs of the computer program ecosystem executed during execution of the computer program ecosystem includes identifying runtime dependencies between the executed computer programs. Identifying runtime dependencies between executed computer programs includes identifying runtime dependencies based on values of parameters in the runtime log information. Analyzing the values of the parameters in the runtime log information includes generalizing one or more of the values. Analyzing the runtime log information includes analyzing values of parameters within the runtime log information to identify one or more data resources accessed during execution of the computer program ecosystem. The method includes analyzing values of parameters within the runtime log information to reveal the name of each data resource accessed during execution of the computer program ecosystem. Identifying the subset of elements as an ecosystem unit includes identifying data resources accessed during execution of the computer program ecosystem as being part of the ecosystem unit.</p><p>Identifying a subset of elements of the computer program ecosystem as an ecosystem unit includes including within the ecosystem unit one or more elements utilized during execution of the computer program ecosystem.</p><p>Performing schedule analysis includes obtaining schedule information from a scheduler, which is separate from the computer program ecosystem.</p><p>Performing a schedule analysis includes identifying a first computer program of the computer program ecosystem that is scheduled to execute after execution of a second computer program of the computer program ecosystem. Identifying the subset of elements as an ecosystem unit includes identifying the first computer program and the second computer program as part of the ecosystem unit.</p><p>The method includes executing a computer program ecosystem to process the data. Executing the computer program ecosystem includes instantiating one or more of the computer programs of the computer program ecosystem according to values of parameters in one or more of parameter sets associated with the computer program ecosystem. . Executing the computer program ecosystem includes executing one or more instantiated computer programs to process data. Runtime log information includes information indicative of one or more instantiated computer programs being executed. Executing the computer program ecosystem includes accessing one or more of the data resources associated with the computer program ecosystem. Runtime log information includes information indicative of one or more accessed data resources. Accessing data resources associated with the computer program ecosystem includes reading data from one or more of files, databases, and objects. Accessing data resources associated with the computer program ecosystem includes writing data to one or more of files, databases, and objects. Executing the computer program ecosystem includes generating runtime log information. Generating the runtime log information includes writing the runtime log information to a log file.</p><p>One or more computer programs include a dataflow graph.</p><p>In one aspect, a computer system for analyzing a computer program ecosystem including a plurality of computer programs includes one or more processors coupled to memory. The one or more processors identify static dependencies between elements of the computer program ecosystem based on values of parameters in one or more parameter sets associated with the computer program ecosystem, Elements of the computer program ecosystem are configured to perform a static analysis of the computer program ecosystem, including identifying a plurality of computer programs of the computer program ecosystem and data resources associated with the computer programs. One or more processors execute a computer program ecosystem, including identifying one or more of the elements of the computer program ecosystem utilized during execution of the computer program ecosystem to process data records. configured to perform temporal analysis; The one or more processors are configured to perform a schedule analysis of the computer program ecosystem including identifying computer programs of the computer program ecosystem that have a schedule dependency on another computer program of the computer program ecosystem. be done. The one or more processors are configured to identify a subset of elements of the computer program ecosystem as ecosystem units based on the results of the static, run-time, and schedule analyses. The one or more processors are configured to migrate the ecosystem unit from the first computer system to the second computer system, test the ecosystem unit, or both.</p><p>In one aspect, a computer-readable medium stores instructions for causing a computer system to analyze a computer program ecosystem including a plurality of computer programs. The instructions are to identify static dependencies between elements of the computer program ecosystem based on the values of parameters in one or more parameter sets associated with the computer program ecosystem. The elements of the computer program ecosystem cause the computer system to perform a static analysis of the computer program ecosystem, including identifying a plurality of computer programs of the computer program ecosystem and the data resources associated with the computer programs. The instructions computer perform run-time analysis of the computer program ecosystem, including identifying one or more of the elements of the computer program ecosystem utilized during execution of the computer program ecosystem to process data records. Let the system do it. The instructions cause the computer system to perform a schedule analysis of the computer program ecosystem including identifying computer programs of the computer program ecosystem that have a schedule dependency on another computer program of the computer program ecosystem. The instructions cause the computer system to identify a subset of the elements of the computer program ecosystem as ecosystem units based on the results of static analysis, runtime analysis, and schedule analysis. The instructions cause the computer system to migrate the ecosystem unit from the first computer system to the second computer system, test the ecosystem unit, or both.</p><p>In one aspect, a computer system for analyzing a computer program ecosystem including a plurality of computer programs analyzes the computer program ecosystem based on the values of parameters in one or more parameter sets associated with the computer program ecosystem. computer program, including identifying static dependencies between elements, wherein the elements of the computer program ecosystem include, identify a plurality of computer programs of the computer program ecosystem and data resources associated with the computer programs; Includes means for performing static analysis of the ecosystem. The computer system performs run-time analysis of the computer program ecosystem, including identifying one or more of the elements of the computer program ecosystem utilized during execution of the computer program ecosystem to process data records. including means for The computer system includes means for performing a schedule analysis of the computer program ecosystem including identifying computer programs of the computer program ecosystem that have a schedule dependency on another computer program of the computer program ecosystem. The computer system includes means for identifying subsets of elements of the computer program ecosystem as ecosystem units based on the results of static analysis, runtime analysis, and schedule analysis. The computer system includes means for migrating the ecosystem unit from the first computer system to the second computer system, testing the ecosystem unit, or both.</p><p>The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will become apparent from the description and drawings, and from the claims.</p>
<figref num="1A">1 is a diagram of a computer program ecosystem; FIG.</figref><figref num="1B">1 is a diagram of a computer program ecosystem; FIG.</figref><figref num="1C">1 is a diagram of a computer program ecosystem; FIG.</figref><figref num="1D">1 is a diagram of a computer program ecosystem; FIG.</figref><figref num="2">FIG. 3 is a diagram of ecosystem unit migration.</figref><figref num="3">FIG. 10 is a diagram of ecosystem unit testing.</figref><figref num="4">1 is a diagram of a system for analyzing a computer program ecosystem; FIG.</figref><figref num="5A">It is a data flow graph.</figref><figref num="5B">5B is a run-time configuration of the dataflow graph of FIG. 5A;</figref><figref num="6">Fig. 3 is a flow diagram for run-time analysis of a computer program ecosystem;</figref><figref num="7">It is a flow chart.</figref><figref num="8">1 is a diagram of a computing system; FIG.</figref>
Techniques for analyzing a computer program ecosystem comprising a plurality of computer programs and associated with a plurality of data resources are described herein to identify subsets of the computer programs and data resources as ecosystem units. A computer program ecosystem is a set of computer programs together with data resources associated with the computer programs. A data resource is data for processing by one or more of the computer programs within the computer program ecosystem that is used as reference data during processing by one or more of the computer programs within the computer program ecosystem (e.g., data records). ), or a destination for data (eg, data records) after processing of the data by one or more of the computer programs in the computer program ecosystem. Ecosystem units are identified based on the results of static, run-time, and schedule analyzes of the computer program ecosystem that indicate dependencies between elements of the computer program ecosystem. System tasks such as migration of computer programs and resources or testing of computer programs can be performed using ecosystem units rather than the entire computer program ecosystem, thereby saving system resources such as processing power or memory. be.
Computer programs and data resources in a computer program ecosystem may have static dependencies on each other. A static dependency between two elements (eg, a computer program and/or a data resource) is a dependency defined by values within a previously stored parameter set associated with the computer program ecosystem. Static dependencies between elements of the computer program ecosystem can be identified by static analysis of values in stored parameter sets.
Computer programs and data resources of the computer program ecosystem may also have dependencies on each other that are not defined until runtime of the computer program ecosystem. When the computer program ecosystem is executed to process the data record, the execution command includes parameter values in addition to or instead of parameter values in previously stored parameter sets that define static dependencies, for example. obtain. Runtime logs generated during execution of the computer program ecosystem can indicate those parameter values, which indicate which computer programs were executed and which data resources were accessed. Runtime analysis, such as analysis of runtime logs, can identify dependencies between elements of the computer program ecosystem that were not apparent from static analysis.
Computer programs in the computer program ecosystem may also have schedule dependencies on each other. Execution of the computer program ecosystem for processing data records may be controlled by a scheduler that defines when particular elements (eg, computer programs) should be executed. In some cases, the scheduler may prescribe that the first computer program or computer programs should be executed after executing the second computer program or computer programs. Schedule analysis, e.g., analysis of scheduling information defined by a scheduler, can identify schedule dependencies between computer programs even if those computer programs have no static or run-time dependencies on each other. can.
By comparing static dependencies, run-time analysis results, and schedule dependencies, a picture of dependencies between elements of the computer program ecosystem can be developed. Elements that depend on or are dependent on another element are identified through a combination of static, run-time, and schedule analysis. Collectively, these identified elements form an ecosystem unit. Treating the computer programs and data resources that form an ecosystem unit as a single entity can facilitate system tasks such as migrating from one computing system to another or testing a computer program ecosystem.
Referring to FIG. 1A, computer program ecosystem 100 includes a plurality of computer programs 102a-102h (collectively referred to as computer programs 102). At least a portion of computer program 102 is statically dependent on one or more other computer programs 102 . For example, computer program 102d accepts as input data (eg, data records) processed by and output from computer program 102a upstream of computer program 102d. Static dependencies are dependencies that can be determined within static analysis prior to run-time of the computer program, for example, based on analysis of stored parameter values or source code associated with the computer program.
In the example of FIG. 1A, computer program 102 of computer program ecosystem 100 is a dataflow graph. A dataflow graph is an executable computer program in graph form that can include executable data processing components and nodes that are data resources such as data sources and data sinks. A data resource is a file, database table, or other type of data source or data source that can, for example, provide data (e.g., data records) for processing by a graph or accept data to be processed by a data processing component of a graph. It can be a data sink. Data processing components and data resources are sometimes collectively referred to as nodes of the graph. An edge connecting two nodes of the graph is provided for the flow of information, such as data or control signals, between nodes. Such dataflow graphs (sometimes referred to as graphs) can be data processing graphs or plans that control the execution of one or more graphs. Dataflow graphs can be implemented to process information. In some examples, one or more data processing components of a dataflow graph may be subgraphs.
One or more data resources 104 a - 104 j (collectively referred to as data resources 104 ) may be associated with computer program ecosystem 100 . A data resource is a data repository, e.g., a source of data processed or used during processing by a computer program ecosystem (e.g., during processing by a computer program of a computer program ecosystem), or of data output by a computer program ecosystem. is the destination. Examples of data resources include files, databases (eg, database tables), queues, objects, and other types of data resources. Accessing a resource means that data is read from or written to the data resource. At least some of the data resources 104 are statically dependent on one or more of the computer programs 102 of the computer program ecosystem 100 or at least some of the computer programs 102 are statically dependent on one or more of the data resources 104 or both. For example, stored parameter values can indicate static dependencies between computer program 102a and data sources 104a, 104f, data from data resource 104a being provided as input to the related computer program 102a, and computer program 102a in turn outputs data records to data resource 104f.
Static dependencies between elements of the computer program ecosystem (e.g., computer programs 102 within computer program ecosystem 100 and data resources 104 associated with computer program ecosystem 100) may be one or more associated with computer program ecosystem 100. Specified by multiple parameter sets. A parameter set can be associated with a particular computer program 102 , a particular resource 104 , or more broadly with the computer program ecosystem 100 . A parameter set can be defined by a file associated with each computer program 102, resource 104, or ecosystem 100, which file contains values (or information that can be used to determine values) for the parameters of the parameter set. )including. As discussed below, static analysis of the computer program ecosystem can be performed to identify static dependencies between elements of the computer program ecosystem based on resolved parameter values of parameters in the parameter set. can.
Referring to FIG. 1B, computer program ecosystem 100 can be executed to process input data, such as input data records. Not all of the elements of computer program ecosystem 100 (eg, computer programs 102 and data resources 104) are necessarily utilized in processing a given set of input data. For example, during execution of computer program ecosystem 100, less than all of computer programs 102 of computer program ecosystem 100 may be executed and/or less than all of data resources 104 associated with computer program ecosystem 100 may be accessed. can be done.
In the example of FIG. 1B, computer programs 102a, 102b, and 102d-102f are executed and data resources 104a, 104b, and 104f are accessed during execution of computer program ecosystem 100 to process input data 110. . Other elements (eg, computer programs 102c, 102g, and 102h and data resources 104c, 104d, 104e, and 104g-104j) are not utilized, as indicated by gray shading in FIG. 1B.
A computer program ecosystem is executed that is responsive to an execution command and can accept parameter values, for example, as arguments supplied to the execution command in addition to or instead of previously stored parameter sets. Static analysis of a computer program ecosystem determines static dependencies based only on stored parameter sets, so parameter sets associated with parameter values derived from arguments supplied to execution commands are not included in static analysis. Not represented in the determined static dependencies. Therefore, identifying ecosystem units based solely on static analysis may result in incomplete or erroneous identification of the elements that make up the ecosystem units.
A run-time analysis may be performed to identify elements of the computer program ecosystem 100 that are utilized during execution of the computer program ecosystem 100, thereby supplementing the computer program ecosystem characterization provided by the static analysis. do. During execution of computer program ecosystem 100, run-time log information is generated that indicates the values of parameters in one or more of the parameter sets associated with computer program ecosystem 100. FIG. Runtime analysis involves analyzing the values of parameters within the runtime log information to identify elements that were utilized during execution of the computer program ecosystem. For example, during execution of a computer program ecosystem, log information written to a file or table may include identification information of the elements of the computer program ecosystem that were executed or accessed. Runtime analysis can interpret the log information to obtain a list of elements executed or accessed during execution.
In some examples, execution of computer program ecosystem 100 may be controlled by a scheduler. In some examples, execution may be scheduled at specific points in time (e.g. a credit card record processing ecosystem may be scheduled to run at 11:00pm nightly, and a call record processing ecosystem may be scheduled to run on Saturday at 2:00 pm). can be scheduled to run at 00am). In some instances, scheduling execution such that one or more computer programs in the computer program ecosystem are scheduled to execute before or after one or more other computer programs in the computer program ecosystem. are called schedule dependencies. In some examples, scheduling dependencies of computer programs 102 may be taken into account in identifying ecosystem units. For example, computer programs with scheduling dependencies may be included in an ecosystem unit even if those computer programs are not identified within the static or run-time analysis of computer program ecosystem 100 .
Referring to FIG. 1C, schedule analysis can identify computer programs within computer program ecosystem 100 that have schedule dependencies on each other. In the example of FIG. 1C, the schedule analysis reveals that computer program 102h has a schedule dependency on computer program 102f, which means that computer program 102h will be executed after computer program 102f has completed its execution. means that
In some examples, schedule analysis can capture dependencies between computer programs 102 that are not reflected by static or runtime analysis. In the example of FIG. 1C, computer program 102h has a schedule dependency on computer program 102f, but no static dependency. Additionally, as shown in FIG. 1B, no runtime dependencies were identified between computer program 102h and computer program 102f. The reason may be that, for example, the scheduling of computer program 102h as running after computer program 102f only concerns certain days of the week and was therefore not captured by runtime analysis.
The results of static, run-time, and schedule analysis of the computer program ecosystem can be used to identify subsets of elements of the computer program ecosystem as ecosystem units. For example, an element used during execution of a computer program ecosystem can be any other element that statically depends on the element that is used, any other element that the element that is used statically depends on, and An ecosystem unit can be formed in addition to any element that shares a scheduling dependency with the Referring to FIG. 1D, within the computer program ecosystem 100, computer programs 102a,b,df,h and resources 104a,b,f form ecosystem units. These computer programs and resources were identified by static and runtime analysis (Figures 1A and 1B) and schedule analysis (Figure 1C). The remaining computer programs 102c,g and resources 104c,e,h,g,j (shaded in gray) are not part of the ecosystem unit.
A user, such as a system developer or system engineer, or a computer-implemented automated operation uses the computer program ecosystem, tests the computer program ecosystem, transfers the computer program ecosystem from one computing system to another computing system. It may be possible to perform system tasks such as migration, or archiving or removing unused code from the computer program ecosystem. To conserve system resources (e.g., processing capacity or memory capacity) while ensuring that all appropriate elements of the computer program ecosystem are addressed within a given system task, the computer program ecosystem as a whole Ecosystem units can be used to perform system tasks rather than using them.
FIG. 2 illustrates an example migration of computer program ecosystem 100 from a first computing system 202 to a second cloud-based computing system 204 . To migrate the computer program ecosystem 100, only those elements of the computer program ecosystem identified as part of the computer program ecosystem ecosystem unit 150 migrate from the first computing system 202 to the second computing system 204. be done. Migrating the entire computer program ecosystem 100 from the first computing system 202 to the second computing system 204 can be a costly process in terms of time, bandwidth, memory, or other factors, for example. Migrating parts of the computer program ecosystem 100 without understanding the relationship between the elements of the computer program ecosystem 100 or the state of each element of the computer program ecosystem 100 during execution is not relevant to a given task. may result in not being able to migrate all the elements that By migrating the ecosystem unit 150 as a whole without migrating elements that do not form part of the ecosystem unit, system resources can be conserved while still allowing all relevant elements of the computer program ecosystem to migrate. .
FIG. 3 shows an example of testing the computer program ecosystem 100 . The computer program ecosystem 100 implements one or more data processing rules (eg, specified and applied by computer programs) each of which acts on at least one input value and produces at least one output value. can be done. whether a given data processing rule is executed during processing of a set of data, e.g. during processing of a given data record, values in each of one or more fields of the data record; It may directly or indirectly depend on values in the set of data.
To test the computer program ecosystem 100, resources such as data sources (e.g., files or tables) are replicated onto the sub-environment 310 from, for example, the server 312 hosting the computer program ecosystem 100, and the computer of the computer program ecosystem 100 is Continue the test in the sub-environment using program instantiation. To more efficiently prepare computer program ecosystem 100 for testing, only resources that are part of ecosystem unit 150 of computer program ecosystem 100 are replicated onto the underlying environment. The testing process can then continue with only the resources relevant to the test replicated.
In the testing process, test data 304 and non-actual production data are provided and processed by computer program ecosystem 100 and test results 306 are output from computer program ecosystem 100 . Test data 304 is data that is specifically prepared or selected for testing and is part or all of the data processing rules within the computer program ecosystem 100 (e.g., the data processing rules of the plurality of computer programs 102 of the computer program ecosystem 100). It may contain enough data to trigger the execution of some or all of the rules. To make testing more efficient and less resource intensive, ecosystem units 150 within computer program ecosystem 100 may be identified prior to testing, and data (eg, of computer programs) within ecosystem units 150 may be identified prior to testing. Sufficient test data 304 to cause some or all of the processing rules to run. Ecosystem unit 150 may be less intensive than full-computer program ecosystem 100, thus reducing the amount and/or variety of test data sufficient to trigger execution of data processing rules within ecosystem unit 150. can be used, resulting in more efficient testing that consumes less processing power.
Referring to FIG. 4, an example analysis system 420 performs static and runtime analysis and schedule analysis on the computer program ecosystem 400 to identify a subset of elements of the computer program ecosystem 400 as ecosystem units 430. . A computer program ecosystem 400 is hosted on a computing system 410 and includes a plurality of computer programs 402 , eg dataflow graphs, and one or more data resources 408 associated with the computer program ecosystem 400 .
Analysis system 420 includes a static analysis engine 422 for identifying static dependencies between elements of computer program ecosystem 400 , such as between computer programs 402 and one or more data resources 408 . The static analysis engine operates by instantiating computer programs 402 of computer program ecosystem 400 and resolving parameters in one or more parameter sets 404 associated with the computer program ecosystem. Static analysis engine 422 then analyzes the resolved parameter values to identify static dependencies between computer programs 402 and data resources 408 . The output of the static analysis process is the identification of static dependencies of computer program ecosystem elements. In some examples, the static analysis process can produce a graphical data tree as output.
One or more runtime logs 406, such as runtime log files, are generated when the computer program ecosystem 400 is executed to process a set of input data, eg, data records. Run-time log 406 includes parameter sets and associated resolved parameter values applicable during execution of computer program ecosystem 400, provided as input to the computer program ecosystem along with, for example, execution commands. Analysis system 420 identifies elements of computer program ecosystem 400 (e.g., computer programs 402 and data resources 408) utilized during execution by analyzing parameter sets and associated resolved parameter values in runtime log 406. includes a runtime analysis engine 424 for
The results of the analysis by static analysis engine 422 and runtime analysis engine 424 are provided to comparison engine 426 to identify the subset of elements of computer program ecosystem 400 that form ecosystem unit 430 . The comparison engine 426 is utilized during execution of all elements of the computer program ecosystem 400 that make up the ecosystem unit 420, e.g., all elements that have a static dependency with another element of the computer program ecosystem. parameter sets and associated resolutions identified by the runtime analysis engine 424 to obtain an identification of all elements that have static dependencies with each of the elements used during execution. It supplements the static dependency identification provided by static analysis by incorporating the parameter values provided.
Schedule analysis engine 428 analyzes schedule dependencies between computer programs 402 in computer program ecosystem 400 , for example by analyzing parameter sets 404 or schedule information 408 associated with scheduler 440 . Also included within ecosystem unit 430 are elements of computer program ecosystem 400 that are identified by schedule analysis engine 428 as having schedule dependencies (eg, with elements already identified for ecosystem units).
Identification of the ecosystem unit 430 can facilitate the execution of system tasks such as testing computer programs or migrating from one computer system to another.
Figures 5A and 5B illustrate example techniques for analyzing static dependencies between elements of a computer program ecosystem. The example approach of FIGS. 5A and 5B includes resolving parameters for an example dataflow graph of a computer program ecosystem and instantiating such dataflow graph. Further discussion of static analysis can be found in US Patent Application Publication No. 2016/0019057, the entire contents of which are incorporated herein by reference.
Referring to FIG. 5A, an example dataflow graph 500 forming part of a computer program ecosystem includes a first subgraph 502 named gather.mp and a second subgraph 504 named process.mp.
The first subgraph 502 accepts as input a first logical data set DS1 506 and a second logical data set DS2 508 and processes data from the first logical data set 506 and the second logical data set 508. , writes the results of the processing into the third logical data set DS3 510. The second subgraph 504 accepts as input a fourth logical data set DS4 512 (which happens to point to the same physical file as the third logical data set 510) and processes data from the fourth logical data set 512. and writes the processing result to the table 514.
Each of the four logical datasets 506, 508, 510, 512 is associated with a parameterized path that resolves to a path to a physical file at runtime. Specifically, the first logical data set 506 was identified using the parameterized path /${FEED}/inv_${DATE}.dat and the second logical data set 508 was parameterized Identified using the path /${FEED}/cust_${DATE}.dat and the third logical data set 510 is identified using the parameterized path /trans_${DATE}.dat and the third Four logical data sets 512 are identified using the parameterized path /trans_${DATE}.dat.
The first subgraph 502 accepts two parameters P1=FEED and P2=DATE as arguments, and the FEED and DATE parameters received FEED and DATE placeholders in the parameterized path as described in more detail below. To resolve the path to the physical location of each of the first logical data set 506, the second logical data set 508, and the third logical data set 510 by replacing the values of their parameters to use. In addition, the first subgraph 502 contains "static analysis" values for the DATE parameter. The static analysis value for the DATE parameter is used as the parameter value during static analysis of dataflow graph 500 (i.e., when the data lineage of dataflow graph 500 is determined), as described in more detail below. is a placeholder value that
Similarly, the second subgraph 504 accepts a single parameter P1=DATE and uses it to receive the DATE placeholder in the parameterized path for the fourth logical data set 512 of the DATE parameter. Resolve the path to the physical location of the fourth logical data set 512 by substituting values. In addition, the second subgraph 504 contains "static analysis" values for the DATE parameter. The static analysis value for the DATE parameter is used as the parameter value during static analysis of dataflow graph 517 (i.e., when the data lineage of dataflow graph 517 is determined), as described in more detail below. is a placeholder value that
Because the behavior of the dataflow graph 500 and its subgraphs is determined by the parameters that the graph accepts, the dataflow graph and its subgraphs are sometimes referred to as "generic" dataflow graphs or "generic" computer programs.
Parameters can be designated as "design-time" parameters or "run-time" parameters. In addition to being used in path resolution as described above, design-time parameters affect the logical operations of their associated dataflow graphs. In contrast, run-time parameters are supplied to the graph on a job-by-job basis and do not affect the logical operations of the graph. In some instances, logical operations in dataflow graphs refer to both the functions of the graph and the logical data sets that the graph utilizes. In some examples, the parameters may be mixed parameters that include both design-time and run-time portions.
In FIG. 5A, the FEED parameter is a design-time parameter that affects the logic operations of the gather.mp subgraph. For example, for one value of the FEED parameter, sort component 516 in first subgraph 502 may sort the data it receives in ascending order, while another different value of FEED parameter causes sort component 516 to sort the data in descending order. obtain. In some examples, a dataflow graph that includes design-time parameters is referred to as a "generic" graph because its logic operations vary based on the supplied values of the design-time parameters.
The DATE parameter is a runtime parameter that does not affect the logical operations of subgraph 502 and is supplied on a job-by-job basis.
In some instances, a set of commonly used parameters for a dataflow graph are stored as a "parameter set" (sometimes called a "pset"), which can be saved to disk for easy reuse. can be done. For example, in FIG. 5A, the first subgraph 502 has three psets associated with it: PSET_mexico 518, PSET_canada 520, and PSET_usa 522. PSET_mexico 518 contains the commonly used FEED parameter value "mexico" and the commonly used DATE parameter value "today()" which is a function that returns today's date. PSET_canada 520 contains the commonly used FEED parameter value "canada" and the commonly used DATE parameter value "today()". PSET_usa 522 contains the commonly used FEED parameter value "usa" and the commonly used DATE parameter value "today()". Similarly, second subgraph 504 is associated with a single pset, PSET 523 . PSET 523 contains a DATE parameter value "today()" which is a commonly used function that returns today's date.
In some examples, prior to executing dataflow graph 500 (eg, as part of executing the computer program ecosystem to which dataflow graph 500 belongs), dataflow graph 500 (and its associated subgraphs 502, 504) may be associated with One or more psets are identified and some specific design-time parameters within the one or more psets are determined. A separate executable instance of the dataflow graph is instantiated for each unique design-time parameter of a given dataflow graph.
For example, referring to Figure 5B, in the dataflow graph 500 of Figure 5B, the first subgraph 502, three instances of gather.mp are instantiated (PSET_mexico->gather.mp502a, PSET_canada->gather.mp502b, PSET_usa-> gather.mp502c), each instance is configured according to a different one of the three unique feed parameters in the pset of Figure 5A: mexico, canada, and usa. Since the second subgraph 504 is associated with only a single pset 523 that does not contain any design-time parameters, only a single instance of the second subgraph 504 (process.mp504a) is instantiated at runtime.
When the appropriate instances of subgraphs 502, 504 are instantiated, the parameter value placeholders in the parameterized path for the dataset are replaced with the actual parameter values from the pset and the physical location of the dataset. Resolve the path to . For example, in the PSET_mexico->gather.mp instance 502a of the first subgraph 502, the FEED parameter value is "mexico" and the DATE parameter value is "031014", so the path for the first dataset 506 is /mexico/inv_031014 resolved to
The dataflow graph 500, including its subgraphs 502, 504, is instantiated and the physical path to the dataset of the dataflow graph is resolved. During execution, the three instances 502a, 502b, 502c of the first subgraph 502 read data from their respective input data sets, process the data, and store the processed data in the /trans_031014.dat physical file. Since the input dataset (e.g. DS4 512) for instance 504a of the second subgraph 502 resolves to the same physical file as the output dataset of the first subgraph, the /trans_031014.dat physical file is It is read, then processed and stored in table 514 .
FIG. 6 illustrates an example approach to runtime analysis of a computer program ecosystem. As noted above, the computer program ecosystem can be executed in response to an execution command that accepts parameter values as arguments. The static analysis of the computer program ecosystem may be incomplete or erroneous, as psets associated with such parameter values are not considered in the static analysis of the computer program ecosystem. Run-time analysis results are used to enhance static analysis results by considering the set of parameters identified in the run-time logs associated with the execution of one or more instances of the computer program ecosystem. can do. Further discussion of runtime analysis can be found in US Patent Application Publication No. 2016/0019057, the entire contents of which are incorporated herein by reference.
In run-time analysis, an example computer program, such as a dataflow graph (eg, subgraph 502 of FIG. 5A), includes two parameters P1 and P2, each of which can be designated as design-time or run-time parameters. Runtime analysis utilizes parameter classifications 626 that designate the classification of each parameter as either a design-time parameter or a runtime parameter. In some examples, runtime analysis may include a parameter classification step 624 . In some examples, the parameters are pre-classified.
Runtime analysis uses a runtime log data store 628, which stores one or more runtime logs each containing information related to the execution of one or more instances of the computer program ecosystem. Including 629. Runtime log 629 may contain a record of executed commands used to instantiate one or more computer programs of the computer program ecosystem for execution. The execute command for a given log entry includes the identification of the computer program and one or more parameter values supplied as arguments to the execute command. In some examples, a computer program is instantiated without access to any parameter set, but instead accepts parameter values as arguments supplied to the execution command.
The parameter classification 626 and the runtime log data store 628 are provided to a processing loop 630 which generates a new logical pset for the execution command for each runtime log 629 in the runtime log data store 628 and creates a new Determines whether the pset already exists in the repository 648 of existing logical psets, and adds the new logical pset to the repository 648 if it does not already exist.
Within processing loop 630, runtime log J from runtime log data store 628<sub>n</sub>632 is analyzed according to parameter classification 626 within logical pset construction step 634 to generate logical pset 636 . In doing so, the logical pset construction step 634 analyzes the execution commands contained within the runtime log 632 to extract the parameter values contained as arguments to the graph execution commands. Logical pset construction step 634 then matches each extracted parameter value with the corresponding parameter in parameter taxonomy 626 . If the logical pset building step 634 determines that the extracted parameter value corresponds to a design-time parameter in the parameter category 626 , the logical pset building step 634 includes the value of the extracted design-time parameter in the logical pset 636 . If logical pset construction step 634 determines that the extracted parameter value corresponds to a run-time parameter in parameter category 626 , then it does not include the extracted parameter value in logical pset 636 .
The identity 644 of the logical pset 636 is determined (642) and compared (646) with the identities of all psets in the repository 648 of existing logical psets. No action is taken if the identification information 644 matches the identification information of at least one existing logical pset in the repository 648, which means that a logical pset for instantiation of the computer program execution command already exists in the repository 648. It is from. If there is no match, an add new logical pset step 650 adds logical pset 636 and its identification 644 as a new logical pset to repository 648 of existing logical psets 648 .
Referring to FIG. 7, an example method for analyzing a computer program ecosystem including a plurality of computer programs performs static analysis of the computer program ecosystem (700). As part of the static analysis, static dependencies between multiple computer programs and data resources associated with the computer program ecosystem are identified (702). For example, static dependencies may be identified based on resolved values of parameters in one or more parameter sets associated with the computer program ecosystem.
Execute the computer program ecosystem to process the data records (704). For example, one or more of the computer programs of the computer program ecosystem are instantiated according to the values of parameters in one or more of the parameter sets associated with the computer program ecosystem and instantiated to process data. A computer program is executed. In some examples, executing the computer program ecosystem accesses one or more of the data resources associated with the computer program ecosystem, e.g., extracts data from one or more of files, databases, and objects. It may include reading, writing data to one or more of files, databases, and objects, or both.
Runtime log information is generated by execution of the computer program ecosystem (706). For example, runtime log information is written to a log file. Runtime log information may include values of parameters in one or more of the parameter sets associated with the computer program ecosystem. For example, run-time log information may be information indicating one or more instantiated computer programs executed during execution of the computer program ecosystem, one or more data accessed during execution of the computer program ecosystem. It may contain information that indicates the resource.
A runtime analysis of the computer program ecosystem is performed (708). Run-time log information is available as part of run-time analysis. Further, as part of the run-time analysis, one of the elements of the computer program ecosystem utilized during execution of the computer program ecosystem, such as computer programs executed or data resources accessed during execution of the computer program ecosystem; The values of the parameters in the runtime log information are analyzed 710 to identify multiples. For example, run-time dependencies between executed computer programs can be identified, for example, based on the values of parameters within the run-time log information. In some examples, analyzing the values of parameters within the runtime log information includes generalizing one or more of the values.
A schedule analysis of the computer program ecosystem is performed to identify computer programs of the computer program ecosystem that have a schedule dependency on another computer program of the computer program ecosystem (712). For example, a schedule analysis can obtain scheduling information from a scheduler and identify computer programs that are scheduled to run after another computer program has completed execution.
A subset of elements of the computer program ecosystem are identified as ecosystem units (714) based on the results of the static analysis, runtime analysis, and schedule analysis of the computer program ecosystem. For example, one or more computer programs executed and one or more data resources accessed have static dependencies on other computer programs or data Contained within an ecosystem unit along with resources.
Execute system tasks using ecosystem units (716). In some examples, the system task includes migrating ecosystem units from a first computing platform to a second computing platform. In some examples, system tasks include testing ecosystem units.
FIG. 8 illustrates an example data processing system 800 for developing and executing dataflow graphs in which the techniques described herein can be used. System 800 includes data sources 802, each of which stores data in any of a variety of formats (eg, database tables, spreadsheet files, flat text files, or native formats used by mainframe computers). may include one or more sources of data, such as a storage device or a connection to an online data stream that may store or provide data. The data can be logistic data, analytical data, or industrial machine data. Execution environment or runtime environment 804 includes preprocessing module 806 and execution module 812 . Execution environment 804 may be hosted on one or more general purpose computers under control of a suitable operating system, such as a version of the UNIX operating system. For example, execution environment 804 may include multiple processing units (central processing units, CPUs, etc.) or processor cores, locally (e.g., multiprocessor systems such as fully symmetric multiprocessing (SMP) computers) or locally distributed (e.g., connected as a cluster). Multiple processors or massively parallel processing (MPP) systems, or remote or remotely distributed (e.g., multiple processors connected via a local area network (LAN) and/or wide area network (WAN)), or any thereof It may include a multi-node parallel computing environment including configuration of computer systems that use combinations.
The storage that provides data source 802 can be local to execution environment 804, for example, stored on a storage medium (e.g., hard drive 808) connected to the computer hosting execution environment 804, or It can be remote, eg, hosted on a remote system (eg, mainframe computer 810) that communicates with the computer hosting execution environment 804 over a remote connection (eg, provided by a cloud computing infrastructure).
Preprocessing module 806 reads data from data source 802 and prepares a data processing application (eg, an executable dataflow graph) for execution. For example, the pre-processing module 806 compiles the data processing application, stores and/or loads the compiled data processing application to and/or from a data storage system 816 accessible to the execution environment 804, and executes the data processing application. Other tasks can be performed to prepare the data processing application.
Execution module 812 executes the data processing application prepared by preprocessing module 806 to process a set of data and generate output data 814 resulting from the processing. The output data 814 may be stored back within the data source 802 or within a data storage system 816 accessible to the execution environment 804, or otherwise used. Data storage system 816 is also accessible to an optional development environment 818 in which developers 820 can design and edit data processing applications to be executed by execution modules 812 . The development environment 818, in some implementations, represents an application as a dataflow graph containing vertices (representing data processing components or datasets) connected by directed edges (representing work elements, i.e., flows of data) between vertices. It is a system for developing For example, such environments are described in Managing Parameters for Graph-Based US Patent Application Publication No. 2007/0011668 entitled "Applications". A system for performing such graph-based computations is described in US Pat. No. 5,966,072, entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," the entire contents of which are incorporated herein by reference . Dataflow graphs created in accordance with the present system provide a way to provide information in and out of the individual processes represented by the graph components, to move information between processes, and to order the execution of processes. do. The system includes an algorithm that selects an interprocess communication method from any of the available methods (e.g. communication paths by graph edges use TCP/IP or UNIX domain sockets to pass data between processes). or use shared memory).
Pre-processing module 806 can accept data from various types of systems that can implement data source 802, including various types of database systems. Data can be organized as records with values for individual fields (also called "attributes" or "columns"), possibly including null values. When data is first read from a data source, the preprocessing module 806 typically starts with some initial format information about the records in that data source. In some situations, the record structure of the data source may not be known initially, and may instead be determined after analysis of the data source or data. Initial information about a record may include, for example, the number of bits that represent distinct values, the order of fields within the record, and the type of value represented by the bits (eg, string, signed/unsigned integer).
In other words, and broadly applicable to executable dataflow graphs described herein, executable dataflow graphs generate one or more data sets from one or more input data sets of data source 802 via data processing components. , the dataflow graph being specified by a data structure in data storage 814, the dataflow graph being specified by a data structure, one or more , which have nodes representing data processing components connected by edges, the edges being specified by the data structure and representing data flow between the data processing components. An execution environment or run-time environment 804 is connected to data storage 814 and hosted on one or more computers to read stored data structures that specify dataflow graphs and to process dataflows by preprocessing module 806 . A pre-processing module 806 configured to allocate and configure system resources (e.g., processes, memory, CPU, etc.) for performing computations of the data processing components specified in the graph, and schedule execution of the computations of the data processing components. and an execution module 812 for controlling. In other words, a runtime environment or execution environment 804 hosted on one or more computers reads data from data sources 802 and processes the data using executable computer programs represented in the form of dataflow graphs. configured to
The techniques described above can be implemented using a computing system running suitable software. For example, software may include one or more computing systems running on one or more programmed or programmable computing systems (which may be of various architectures such as distributed, client/server, or grid). Such computing system may include procedures in computer programs, and such computing system includes at least one processor, at least one data storage system (including volatile and/or non-volatile memory and/or storage elements), (at least one input each includes at least one user interface (for receiving input using a device or port and for output using at least one output device or port). Software may include, for example, one or more modules of a larger program that provide services related to graph design, construction, and execution. Program modules (eg, graph elements) may be implemented as data structures or other organized data conforming to a data model stored in a data repository.
The software may be provided on, or executed on, a tangible, non-transitory medium such as a CD-ROM or other computer-readable medium (readable by, for example, a general-purpose or special-purpose computing system or device). It can be delivered over the communication medium of the network to the tangible, non-transitory medium of the computing system (eg, encoded within the propagating signal). Some or all of the processing may be performed on a special purpose computer or using special purpose hardware such as a coprocessor or programmable gate array (FPGA) or special application specific integrated circuit (ASIC). Processing may also be implemented in a distributed fashion, where different parts of the computation specified by the software are performed by different computational elements. Each such computer program is preferably accessible by a general purpose or special purpose programmable computer to configure and operate the computer when the storage medium is read by the computer to perform the processes described herein. computer readable storage medium (eg, solid state memory or media or magnetic or optical media) in any storage device, or downloaded to such computer readable storage medium. The system of the present invention may also be considered to be implemented as a tangible, non-transitory medium composed of a computer program, which medium may be implemented in one or more of the processing steps described herein. operate a computer in a specific and predetermined manner to execute
A number of embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the invention. For example, some of the steps above may be order independent and thus may be performed in a different order than described.
Other implementations are within the scope of the appended claims.
100 computer program ecosystem
102 computer program
104 data resources
150 ecosystem unit
202 First computing system
204 Second computing system
304 test data
306 Test results
310 Sub-environment
312 server
400 computer program ecosystem
402 computer program
404 parameter set
406 runtime log file
408 Data resource (schedule information)
420 Analysis system
422 static analysis engine
424 runtime analysis engine
426 comparison engine
428 schedule analysis engine
430 ecosystem unit
440 scheduler
502 first subgraph
502a,502b,502c instances of the first subgraph
504 second subgraph
504a second subgraph instance
506 First Logical Dataset (DS1)
508 Second Logical Dataset (DS2)
510 Third Logical Dataset (DS3)
512 Fourth Logical Dataset (DS4)
514 table
516 sort component
517 data flow graph
518,520,522,523 parameter sets
800 data processing system
802 data source
804 execution environment or runtime environment
806 Pretreatment module
808 hard drive
810 mainframe computer
812 execution module
814 output data
816 data storage system
818 Development environment
820 Developer
12 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP2019046001A | Cites | Japan |
| JP2018116517A | Cites | Japan |
| JP2013156802A | Cites | Japan |
| JP2011060277A | Cites | Japan |
| JP2009245066A | Cites | Japan |
| US20110258611A1 | Cites | United States of America |
| US20130219057A1 | Cites | United States of America |
18 members in 11 offices
Priority claims5
| Document | Office | Kind | Date |
|---|---|---|---|
| 62851295 | United States of America | – | |
| 201962851295 | United States of America | P | |
| 16746392 | United States of America | – | |
| 202016746392 | United States of America | A | |
| 2020033998 | United States of America | W |
Members18
| Document | Office | Kind | |
|---|---|---|---|
| US2020371781A1 | United States of America | A1 | |
| WO2020237051A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US10996946B2 | United States of America | B2 | |
| US2021263734A1 | United States of America | A1 | |
| AU2020279774A1 | Australia | A1 | |
| CA3141609A1 | Canada | A1 | |
| SG11202112004VA | Singapore | A | |
| CN114207594A | China | A | |
| DE112020002515T5 | Germany | T5 | |
| EP3973400A1 | European Patent Office (EPO) | A1 | |
| BR112021023350A2 | Brazil | A2 | |
| JP2022525250A | Japan | A | |
| AU2020279774B2 | Australia | B2 | |
| US11487534B2 | United States of America | B2 | |
| JP7204011B2This record | Japan | B2 | |
| CN114207594B | China | B | |
| ZA202110741B | South Africa | B | |
| EP3973400B1 | European Patent Office (EPO) | B1 |
10 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 | |
| Explanation of circumstances concerning accelerated examinationJAPANESE INTERMEDIATE CODE: A871A871 | A871 |
Numbers
- Publication
- 7204011
- Application
- 2021569170
Titles2
- Japanese
- コンピュータプログラムシステムの静的及び実行時分析
- English
- Static and runtime analysis of computer program systems
Classification
- CPC, 11
- G06F11/3688
- G06F8/75
- G06F11/3612
- G06F11/3608
- G06F11/3676
- G06F11/323
- G06F11/3698
- G06F11/3476
- G06F11/362
- G06F8/71
- G06F8/60
- IPC, 1
- G06F11 36
