go-warc

A golang library to work with WARC files from the common crawl

Stars

15

Forks

7

Language

Go

Last Updated

Feb 15, 2023

Similar Repos
Repo Language Stars Description Updated At
Python 2 Scripts to verify Common Crawl segments and WARC/WET/WAT files Sep 23, 2021
Go 11 warc library for golang Feb 04, 2019
Shell 3 Sample code to grep Common Crawl WARC files in Go, Java, Node and Python. Feb 10, 2022
C++ 470 Common Crawl support library to access 2008-2012 crawl archives (ARC files) Aug 21, 2022
Rust 6 builds a tantivy index from common crawl warc.wet files May 15, 2022
Haskell 5 A Pipes-based parser for the Web Archive (WARC) format used by the Common Crawl and … Jan 05, 2022
Python 21 Gathers urls from common crawl Jun 14, 2022
C++ 11 C++ library to parse WARC files Feb 11, 2022
Java 6 Extract text information from warc files Mar 23, 2022
Python 6 Generate WARC files from dynamic webpages Apr 08, 2023
Java 2 playing around with the common crawl dataset Jan 04, 2013
Python 54 Statistics of Common Crawl monthly archives mined from URL index files Aug 03, 2022
Lex 2 Extract URL's from Common Crawl data Mar 18, 2022
Go 22 Miscellaneous tools for processing WARC files from the CommonCrawl May 17, 2021
Go 3 golang common library Apr 23, 2023
Go 5 Using Golang to crawl data from Stackoverflow Sep 29, 2020
Python 6 Python library for reading and writing warc files Mar 05, 2016
Python 223 Python library for reading and writing warc files Aug 07, 2022
Python 2 Python library for reading and writing warc files Jan 08, 2013
Python 3 Python library for reading and writing warc files May 30, 2019
Rust 3 Extract multimodal training data from Flickr WARC files May 03, 2022
Python 10 Support for writing WARC files with Scrapy Feb 01, 2022
Go 2 Work with .gma (Garrys Mod Addon) files from within your GoLang application Feb 12, 2023
Java 4 Application for downloading text data from Common Crawl Sep 07, 2021
Java 32 Java library for reading and writing WARC files with a typed API Mar 07, 2022
None 3 Common gotchas with golang and how to work around them Jul 21, 2022
Python 19 Python 3 library for reading and writing warc files Mar 23, 2023
Shell 18 Useful tools to extract malayalam text from the Common Crawl Datasets Oct 04, 2022
JavaScript 166 Chrome extension to "Create WARC files from any webpage" Oct 09, 2022
Go 2 golang library for scaffolding files from templates Jan 20, 2023
Python 76 Search the common crawl using lambda functions Mar 27, 2023
Python 23 Extract data from common crawl using elastic map reduce Dec 08, 2021
Python 234 Process Common Crawl data with Python and Spark Aug 29, 2022
Go 3 Library to work with Pyrus written in Golang Feb 17, 2023
None 9 Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries … Jan 06, 2022
JavaScript 86 Parse And Create Web ARChive (WARC) files with node.js May 16, 2023
Go 50 Common utilities for dealing with Apple Software files in Golang. Nov 26, 2022
Shell 22 Tools to construct and process webgraphs from Common Crawl data Aug 23, 2022
Go 32 🕸 A simple way to extract data from Common Crawl Jun 13, 2022
Python 2 Python tool for extracting the plain text and hyperlinks from Common Crawl database. Nov 22, 2019
Go 2 Common golang library for Prometheus exporters Nov 02, 2022
Go 4 Common Golang library for my projects Jun 08, 2021
Go 23 Go library to work with Parquet Files Jun 18, 2022
Go 2 Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl. Apr 05, 2023
Go 2825 Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl. May 02, 2023
None 2 Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl. May 12, 2023
C++ 22 Several sketches and library files that work with the ATTiny85 May 02, 2022
JavaScript 8 Crawl data from the AppStore Oct 01, 2022
Python 3 Parsing Huge Web Archive files from Common Crawl data index to fetch any required domain's … Apr 07, 2022
C# 2 Fast and easy library for reading and writing WARC (Web Archive) files Sep 10, 2023