PiFlow是一个基于分布式计算框架Spark开发的大数据流水线系统。该系统将数据的采集、清洗、计算、存储等各个环节封装成组件,以所见即所得方式进行流水线配置。简单易用,功能强大。它具有如下特性:

  1. 简单易用:可视化配置流水线,实时监控流水线运行状态,查看日志
  2. 功能强大:提供100+的数据处理组件, 包括Hadoop 、Spark、MLlib、Hive、Solr、Redis、MemCache、ElasticSearch、JDBC、MongoDB、HTTP、FTP、XML、CSV、JSON等,同时集成了微生物领域的相关算法。
  3. 扩展性强:支持自定义开发数据处理组件
  4. 性能优越:基于分布式计算引擎Spark开发

试用地址:http://piflow.ml/piflow-web/login ,用户名/密码:admin/admin

安装使用说明详见:https://github.com/cas-bigdatalab/piflow

支持的数据处理组件如下:

组名

组件名

Hive:

SelectHiveQL,PutHiveStreaming,PutHiveQL

HDFS

PutHdfs,DeleteHdfs,UnzipFilesOnHDFS,GetHdfs,ListHdfs,SelectFilesByName

HTTP

InvokeUrl,GetUrl,UnGZip,PostUrl,LoadZipFromUrl,FileDownHDFS

RDF

CsvToNeo4J,RdfToDF

Spider

Spider

JDBC

JdbcRead,JdbcReadFromOracle,JdbcWrite,JdbcWriteToOracle,SelectImpala

Streaming

FlumeStream,KafkaStream,SocketTextStreamByWindow,SocketTextStream,TextFileStream

MongoDB

GetMongo,PutMongo

CSV

FolderCsvParser,CsvSave,CsvParser,CsvStringParser

File

PutFile,FetchFile,RegexTextProcess

Script

ShellExecutor,DataFrameRowParser

Common

Distinct,ConvertSchema,Fork,SelectField,Join,DoFlatMapStop,ExecuteSQLStop,Merge,DoMapStop,Subtract

Data Clean

IdentityNumberClean,PhoneNumberClean,EmailClean,TitleClean

Message Queue

WriteToKafka,ReadFromKafka

Microorganism

Ensembl_gff3Parser,GeneParser,RefseqParser,GoDataParse,PfamDataParser,GoldDataParse,Swissprot_TrEMBLDataParser,EmblParser,PDBParser,GenBankParse,TaxonomyParse,BioProjetDataParse,BioSampleParse,InterprodataParse,MicrobeGenomeDataParser

MemCache

ComplementByMemcache,PutMemcache,GetMemcache

Machine Learning

cGaussianMixtureTraining,LogisticRegressionTraining,RandomForestTraining,DecisionTreePrediction,RandomForestPrediction,BisectingKMeansPrediction,LDAPrediction,MultilayerPerceptronTraining,GBTTraining,BisectingKMeansTraining,MultilayerPerceptronPrediction,GBTPrediction,KmeansTraining,NaiveBayesPrediction,DecisionTreeTraining,LDATraining, WordToVec ,LogisticRegressionPrediction,KmeansPrediction,NaiveBayesTraining,GaussianMixturePrediction

ElasticSearch:

PutEs,QueryEs,FetchEs

Redis:

WriteToRedis,ReadFromRedis

XML

XmlParser,XmlStringParser,FlattenXmlParser,XmlSave,FolderXmlParser

FTP

UploadToFtp,LoadFromFtpToHDFS

Excel

ExcelParser

Solr

PutIntoSolr,GetFromSolr

JSON

JsonStringParser,MultiFolderJsonParser,FolderJsonParser,JsonSave,JsonParser,EvaluateJsonPath,

GraphX

LoadGraph,LabelPropagation