Data extraction, Data scraping, Web scraping, Scraper, Extract, Transform, Load, Data analytics
Платформа:
Windows
ОС:
Win2000, WinXP, Win7 x32, Win7 x64, Windows 8, Windows 10, WinServer, WinOther, WinVista, WinVista x64
Системные требования:
Windows x64
Язык:
English, Russian
Ограничения:
No limitations
Ключевые слова: Диггер, Диггернаут, скрапер, парсер, веб парсинг, веб скрапинг, сбор информации, извлечение данных, аналитика, биг дата, бизнес разведка, ETL
Диггернаут - это облачный сервис для парсинга сайтов, сбора информации и других ETL (Extract, Transform, Load) задач. Если ваш бизнес лежит в плоскости торговли и ваш поставщик не предоставляет вам данные в нужном вам формате, например в csv или excel, мы можем вам помочь избежать ручной работы, сэкономив ваши время и деньги!
Все, что вам нужно сделать - создать парсер (диггер), крошечного робота, который будет парсить сайты по вашему запросу, извлекать данные, нормализовать и обрабатывать их, сохранять массивы данных в облаке, откуда вы сможете скачать их в любом из доступных форматов (например, CSV, XML, XLSX, JSON) или забрать в автоматическом режиме через наш API.
Если вы никогда не сталкивались с программированием, вы можете использовать наш специальный инструмент для построения конфигурации парсера (диггера) - Excavator. Он имеет графическую оболочку и позволяет работать с сервисом людям, не имеющих теоретических познаний в программировании. Вам нужно лишь выделить данные, которые нужно забрать и разместить их в структуре данных, которую создаст для вас парсер. Для более простого освоения этого инструмента, мы создали серию видеоуроков, с которыми вы можете ознакомиться в документации.
Если вы программист или веб-разработчик, знаете что такое HTML/CSS и готовы к изучению нового, для вас мы приготовили мета-язык, освоив который вы сможете решать очень сложные задачи, которые невозможно решить с помощью конфигуратора Excavator. Вы можете ознакомиться с документацией, которую мы снабдили примерами из реальной жизни для простого и быстрого понимания материала.
Ключевые слова: Digger, Diggernaut, data, web scraping, data scraping, data extraction, data analytics, big data, data visualisation, scraping, API, narrator, ETL
Diggernaut is a cloud based service for web scraping, data extraction and other ETL (Extract, Transform, Load) tasks. If you are a reseller of goods and your supplier does not let you have their data in a suitable format, such as excel or csv, you will have to manually retrieve data from their website. All you need to do is to create a digger, a tiny robot that can do web scraping on your behalf and extract data from websites for you, normalize it and save data to the cloud. Once its done, you can download it in CSV, XLS, JSON format or even retrieve it using our Rest API.
If you dont have any programming skills, you can use Excavator tool to build configuration for your scrapers. Its simple point & click application where you select what data you want to extract and set how you want to have it in your output data structure. We have video lessons explaining how to use it on our sandbox website.
If you are a programmer or web-developer, you can use meta-language to build configuration for a digger. This way you can solve really complex data extraction tasks. We have comprehensive meta-language documentation which helps you to get used into it; also we have many posts on our blog where we show how you can use different meta-language functions to solve different real cases.
Разрешения на распространение: Any versions of Diggernaut's Excavator tool can be freely distributed over the internet in an unchanged form.
Terms of Service Agreement
PLEASE READ THE FOLLOWING TERMS OF SERVICE AGREEMENT CAREFULLY. BY ACCESSING OR USING OUR WEBSITES AND OUR SERVICES, YOU HEREBY AGREE TO BE BOUND BY THE TERMS AND ALL TERMS INCORPORATED HEREIN BY REFERENCE. IT IS THE RESPONSIBILITY OF YOU, THE USER, CUSTOMER, OR PROSPECTIVE CUSTOMER TO READ THE TERMS AND CONDITIONS BEFORE PROCEEDING TO USE THIS WEBSITE. IF YOU DO NOT EXPRESSLY AGREE TO ALL OF THE TERMS AND CONDITIONS, THEN PLEASE DO NOT ACCESS OR USE OUR WEBSITES OR OUR SERVICES. THIS TERMS OF SERVICE AGREEMENT IS EFFECTIVE AS OF 06/15/2016. Acceptance of Terms
The following Terms of Service Agreement (the 'TOS') is a legally binding agreement that shall govern the relationship with our users and others which may interact or interface with Diggernaut LLC (the 'Diggernaut'), located at 4620 Greenore Dr, High Ridge, Missouri 63049 and our subsidiaries and affiliates, in association with the use of the website, which includes Diggernaut, (the 'Website') and its Services, which shall be defined below.
Diggernaut may at any time modify these Terms, and your continued use of this Website and the Services will be conditioned upon the terms and conditions in force at the time of your use. Your registration for, or use of, the Services shall be deemed to be your approval and execution of this Agreement including your covenant to abide by the TOS including those related to any materials available on the Website. Services
Diggernaut shall reserve the right at any time it may deem fit, to modify, alter and or discontinue, whether temporarily or permanently, our service, or any part thereof, with or without prior notice. In addition, we shall not be held liable to you or to any third party for any such alteration, modification, suspension and/or discontinuance of our Services, or any part thereof. Uses
You agree that your use of the technologies, tools, applications, and products made available from time to time on the Website (as further defined in these TOS, the 'Services') are used solely in conjunction with the extraction of data from publicly accessible websites (including sites which may require a user login). In all cases where copyrighted data may be included as part or all of the extracted data, you agree to comply with all copyright laws and rules covering such data including (but not limited to) the reproduction and resale of such data. You agree that Diggernaut will not be held responsible for legal liabilities incurred through the use or misuse of data obtained through the Services.
You are responsible for all activity occurring under your user account(s) and shall abide by all applicable local, state, national and foreign laws, treaties and regulations in connection with your use of the Services, including those related to data privacy, international communications and the transmission of technical or personal data. You shall:
- Notify Diggernaut immediately of any unauthorized use of any password or account or any other known or suspected breach of security - Report to Diggernaut immediately and use reasonable efforts to stop immediately any copying or distribution of Content or infringement of Diggernaut Technology that is known or suspected by you or your Users - Not impersonate another Diggernaut user or provide false identity information to gain access to or use the Services
You shall not use any of the Services or this Website:
- to extract data from websites that restricts automated data extraction in their TOS, unless you have permission from websites owners - for any fraudulent activity - for any extraction of data which is contrary to applicable laws or third party rights (including Intellectual Property) Rights) - to engage in any other conduct which is contrary to applicable laws or third party rights (including Intellectual Property Rights) or which constitutes a fraudulent business practice under any applicable laws - in any way that causes, or may cause, damage to the website or impairment of the availability or accessibility of the website; or in any way which is unlawful, illegal, fraudulent or harmful, or in connection with any unlawful, illegal, fraudulent or harmful purpose or activity - to copy, store, host, transmit, send, use, publish or distribute any material which consists of (or is linked to) any spyware, computer virus, Trojan horse, worm, keystroke logger, rootkit or other malicious computer software - to transmit or send unsolicited commercial communications
License Grant and Restrictions
Diggernaut hereby grants you a non-exclusive, non-transferable, worldwide right and license to use the Services, solely for your own internal purposes, subject to the Terms of this Agreement. All rights not expressly granted to you are reserved by Diggernaut and its licensors.
You may not access the Services if you are a direct competitor of Diggernaut. In addition, you may not access the Services for purposes of monitoring its availability, performance or functionality, or for any other benchmarking or competitive purposes. Any action of this kind is likely to result in serious harm to Diggernaut for which you agree that Diggernaut will be entitled to immediate injunctive relief together with other legal and equitable remedies.
You shall not:
- License, sublicense, sell, resell, transfer, assign or distribute the Services in any way other than as permitted in writing by Diggernaut - Modify or make derivative works based upon the Services - Modify, reverse engineer, decompile, disassemble or access any computer code relating to the Services in order to: - build a competitive product or service - build a product using similar ideas, features, functions or graphics of the Services - copy any ideas, features, functions or graphics of the Services
User Submitted Data
The Website allows you and other users to import, export and use data from third party websites and other sources. This data is referred to herein as 'User Content.'
You are solely responsible for User Content and the consequences of Diggernaut or any third party importing, exporting or using such User Content. In connection with any such User Content, you affirm, represent, and/or warrant that you own or have the necessary licenses, rights, consents, and permissions related to the User Content and its use by you, kimono labs, and any other third party
In connection with User Content, you further agree that you will not:
- submit material that is copyrighted, protected by trade secret or otherwise subject to third party proprietary rights, including privacy and publicity rights, unless you are the owner of such rights or have permission from their rightful owner to post the material and to grant Diggernaut all of the rights and licenses granted herein - submit material that is unlawful, obscene, defamatory, libelous, threatening, pornographic, harassing, hateful, racially or ethnically offensive, or encourages conduct that would be considered a criminal offense, give rise to civil liability, violate any law, or is otherwise inappropriate - post advertisements or solicitations of business - impersonate another person
Diggernaut does not endorse any User Content or any data, opinion, recommendation, or advice expressed therein, and it expressly disclaims any and all liability in connection with any User Content. Diggernaut does not permit copyright infringing activities or infringement of intellectual property rights on the Website, and will remove all Content and User Content if properly notified that such Content or User Content infringes on another's intellectual property rights. Diggernaut may remove or choose not to publish any Content or User Content without prior notice. Diggernaut may also terminate your access to the Website, if you are determined to be a repeat infringer. A repeat infringer is anyone who has been notified of infringing activity more than once and/or has had any User Content removed from the Website more than twice. Diggernaut also reserves the right to decide whether Content or User Content is appropriate and complies with this Agreement for violations other than copyright infringement and violations of intellectual property law, such as, but not limited to, pornography, obscene or defamatory material. Diggernaut may remove any such User Content and/or terminate your access for uploading such material in violation of this Agreement at any time, without prior notice and at its sole discretion. Legal Disclaimer
As a service provider, in the event that you have a dispute with a data source (target where you extract data from) in relation to the use (or misuse) of any of the Services, you release the Website (and Diggernaut LLC. and our officers, directors, agents, subsidiaries, joint ventures and employees) from any and all claims, demands and damages (actual and consequential) of every kind and nature, known and unknown, suspected and unsuspected, disclosed and undisclosed, arising out of or in any way connected with such disputes. If you are a California resident, you waive California Civil Code 1542, which says: 'A general release does not extend to claims which the creditor does not know or suspect to exist in his favor at the time of executing the release, which if known by him must have materially affected his settlement with the debtor.
Diggernaut assumes that the you will use Services legally and ethically and that you have obtained permission to use it on the targeted data source, if necessary. Diggernaut reserves the right to refuse service to anyone wishing to use the Services in an illegal manner.
Diggernaut may immediately suspend your use of the Services if it is contacted by your chosen data extraction source, website, target or repository and asked to cease all extraction activity. If such an event occurs, Diggernaut will not disclose your information without a court order mandating it to do so. If however Diggernaut receives a court order demanding the release of your information to a third party, it will comply. If such action becomes necessary, you agree to indemnify and hold Diggeranut and (as applicable) its parent(s), subsidiaries, affiliates, officers, directors, agents, contractors and employees, harmless from any claim or demand, including reasonable attorneys' fees, made by any third party arising from any complaint, suit, disagreement or other repercussions resulting from your use of the Services. Customer Data and Confidentiality
Diggernaut does not own any data, information or material that you submit to the Services in the course of using the Services ('Customer Data') and may not use or disclose Customer Data other than:
- to comply with law or comply with legal process - to protect and defend Diggernaut's rights or the rights of Diggernaut's Customers and third parties - to avoid liability or the liability of Diggernaut Customer's or third parties
Diggernaut may gather and use any information describing the habits, usage patterns or demographics relating to any Customer Data.
You shall have sole responsibility for the accuracy, quality, integrity, legality, reliability, appropriateness, and intellectual property ownership or right to use of all Customer Data, and Diggernaut shall not be responsible or liable for the deletion, correction, destruction, damage, loss or failure to store any Customer Data.
During the term of your use of the Services and for a period of two (2) years following the termination or expiration of your subscription, each party agrees not to disclose Confidential Information of the other party to any third party without prior written consent except as provided herein. 'Confidential Information' includes subscriptions, including their definition and content and any other Services information or access to technology prior to public disclosure provided by Diggernaut to you and identified at the time of disclosure in writing as 'Confidential.'
Confidential Information does not include information that has become publicly known through no breach by a party, or has been independently developed without access to the other party's Confidential Information, rightfully received from a third party or required to be disclosed by law or by a governmental authority. Nothing in these TOS shall prohibit or limit either party's use or disclosure of the U.S. Federal income tax treatment and U.S. Federal income tax structure of any transaction contemplated by these Terms and all materials of any kind (including opinions or other tax analyses) that are provided to it relating to such tax treatment or tax structure, except where confidentiality is necessary to comply with applicable federal or state securities laws. Cancellation. Refund.
You may cancel your subscription at any time but shall remain liable for billing period you already had been changed for. It's your responsibility to cancel your subscription before billing date if you do not want to be charged. No refunds will be given for un-used page requests or remaining monthly bandwidth at time of cancellation. Termination.
Diggernaut may at any time terminate your use of the Services or cancel any subscription(s) or your use of any subscription. Except as set forth above or unless Diggernaut has previously canceled or terminated your use of the Services (in which case subsequent notice by Diggernaut shall not be required), Diggernaut will notify you via email of any such termination, which shall be effective immediately. Upon termination or expiration of this Agreement for any reason, you shall remain liable for any amount due for subscriptions already delivered through the date such termination takes effect. Warranty.
DIGGERNAUT MAKES NO REPRESENTATION, WARRANTY, OR GUARANTY AS TO THE RELIABILITY, TIMELINESS, QUALITY, SUITABILITY, TRUTH, AVAILABILITY, ACCURACY OR COMPLETENESS OF THE SERVICES OR ANY CONTENT. DIGGERNAUT DOES NOT REPRESENT OR WARRANT THAT (A) THE USE OF THE SERVICES WILL BE SECURE, TIMELY, UNINTERRUPTED OR ERROR-FREE OR OPERATE IN COMBINATION WITH ANY OTHER HARDWARE, SOFTWARE, SYSTEM OR DATA, (B) THE SERVICES WILL MEET YOUR REQUIREMENTS OR EXPECTATIONS, (C) ANY STORED DATA WILL BE ACCURATE OR RELIABLE, (D) THE QUALITY OF ANY PRODUCTS, SERVICES, INFORMATION, OR OTHER MATERIAL PURCHASED OR OBTAINED BY YOU THROUGH THE SERVICES WILL MEET YOUR REQUIREMENTS OR EXPECTATIONS, (E) ERRORS OR DEFECTS WILL BE CORRECTED, OR (F) THE SERVICES OR THE SERVER(S) THAT MAKE THE SERVICES AVAILABLE ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS. THE SERVICES AND ALL CONTENT IS PROVIDED TO YOU STRICTLY ON AN 'AS IS' BASIS. ALL CONDITIONS, REPRESENTATIONS AND WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT OF THIRD PARTY RIGHTS, ARE HEREBY DISCLAIMED TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW BY DIGGERNAUT. Limitation of Liability.
EXCEPT FOR ANY INDEMNIFICATION AND CONFIDENTIALITY OBLIGATIONS HEREUNDER, IN NO EVENT SHALL EITHER PARTY BE LIABLE UNDER THESE TERMS FOR ANY CONSEQUENTIAL, SPECIAL, INDIRECT, EXEMPLARY, OR PUNITIVE DAMAGES WHETHER IN CONTRACT, TORT OR ANY OTHER LEGAL THEORY, EVEN IF SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY AND DIGGERNAUT'S AGGREGATE LIABILITY TO YOU UNDER THIS AGREEMENT FOR ANY CLAIM IS LIMITED TO THE AMOUNT PAID TO DIGGERNAUT BY YOU FOR THE SERVICES GIVING RISE TO THE CLAIM. Each party acknowledges that the other party has agreed to these Terms relying on the limitations of liability stated herein and that those limitations are an essential basis of the bargain between the parties. Without limiting the foregoing and except for payment obligations, neither party shall have any liability for any failure or delay resulting from any condition beyond the reasonable control of such party, including but not limited to governmental action or acts of terrorism, earthquake or other acts of God, labor conditions, and power failures. Representations and Warranties.
You represent and warrant that (a) all of the information provided by you to Diggernaut to use the Services is correct and current and that that you have not falsely identified yourself nor provided any false information to gain access to the Services and that your billing information is correct. Your Obligation to Indemnify.
You agree to indemnify, defend and hold Diggernaut, its agents, affiliates, subsidiaries, directors, officers, employees, and applicable third parties (e.g., all relevant Partner(s), licensors, licensees, consultants and contractors) ('Indemnified Person(s)') harmless from and against any third party claim, liability, loss, and expense (including damage awards, settlement amounts, and reasonable legal fees), brought against any Indemnified Person(s), arising out of your use of the Services and/or your breach of any of these Terms. You acknowledge and agree that each Indemnified Person has the right to assert and enforce its rights under this Section directly on its own behalf as a third party beneficiary. Miscellaneous.
You will be responsible for all reasonable expenses (including attorneys' fees) incurred by Diggernaut in collecting unpaid amounts under these TOS. These TOS shall be governed by the laws of the State of Missouri, except for its conflicts of laws principles. Any dispute or claim arising out of or in connection with these TOS shall be adjudicated in Missouri.