08 Gtu TPT Report
08 Gtu TPT Report
Trusted Microsoft Partner for Dynamics and Cloud Solutions, We are one of India’s top
Microsoft Dynamics 365 partners. Our team has more than 21 years of experience and
more than 9 business verticals of expertise, which makes us exceptional for ERP and
CRM solution providers nationwide. Dynamics 365 deployment is making the lives of
Small and Mid-sized Businesses much simpler.
Mission:
Allow businesses to see through their balance sheets and dissect deep into the statistics in
each aspect of their business.
Value:
Timely delivery of the solutions. Standing by the side of the clients before, while and
after the delivery.
JOINING LETTER
COMPLETION CERTIFICATE
ACKNOWLEDGEMENT
First I would like to thank Mr. Harsh Makwana, Director and Miss. Foram
Vithalani, HR of Tripearltech Pvt Ltd. for giving me the opportunity to do an
internship within the organization.
I also would like all the people that worked along with me in the
organization with their patience and openness to create an enjoyable
working environment.
I am highly grateful to Principal Dr. Chirag Vibhakar Sir , for the facilities
provided to accomplish this internship.
I would like to thank my Head of the Department Prof. Mahesh Prajapati, for
the constructive criticism throughout my internship.
Data mining is the process of sorting through large data sets to identify
patterns and relationships that can help solve business problems through data
analysis. Data mining techniques and tools enable enterprises to predict
future trends and make more-informed business decisions.
Web scraping is the process of using bots to extract content and data from a
website.
Low code Power Automation makes work very easier as already basic code
is provided in the respective software.
Data Science and analysis is playing the most significant role today covering
every industry in the market. For e.g., finance, e-commerce, business,
education, government.
And so I learned the easiness of how I can make a low code application
rather than writing whole code from scratch and using the inbuilt code to
accomplish something more creative and innovative out of that.
Netflix is a pure data science project that reached at the top through
analyzing every single interest of their customers. Key terminology that are
used in Data Science are: Data Visualization, Anaconda Jupyter Notebook,
Exploratory Data Analysis, Machine Learning, Data wrangling, and
Evaluation using scikit library’s surprise module
DAY – 1:
BASIC INTRODUCTION AND DOMAIN KNOWLEDGE!
● Data Mining :
Data Mining is a crucial component of successful analytics initiatives in
organizations. The information it generates can be used in Business
Intelligence (BI) and advanced analytics applications that involve analysis of
historical data, as well as Real time analytics applications that examine
streaming data as it's created or collected.
As there are there are so many ways that we can do data mining there are
two main ways to do data mining:
1. Classification Analysis
2. Regression Analysis
3. Time Serious Analysis
4. Prediction Analysis
1. Classification Analysis
2. Regression Analysis
3. Time Serious Analysis
4. Prediction Analysis
I researched a lot regarding the steps of data mining and scraping of the data
to get a lot of uniform data.
● Web Scraping :
Web scraping is the process of using bots to extract content and data from a
website.
Unlike screen scraping, which only copies pixels displayed onscreen, web
scraping extracts underlying HTML code and, with it, data stored in a
database. The scraper can then replicate entire website content elsewhere.
● Market research companies using scrapers to pull data from forums and
social media (eg: for sentiment analysis).
Web scraping is also used for illegal purposes, including the undercutting of
prices and the theft of copyrighted content. An online entity targeted by a
scraper can suffer severe financial losses, especially if it’s a business
strongly relying on competitive pricing models or deals in content
distribution.
Web scraping has countless applications, especially within the field of data
analytics. Market research companies use scrapers to pull data from social
media or online forums for things like customer sentiment analysis. Others
scrape data from product sites like Amazon or eBay to support competitor
analysis.
Tools for scraping the data:
Python is more favorable for scraping the data because it has so many builtin
libraries like BeautifulSoup, Scrapy, Parsehub and so on. We can also use
Selenium because it is better to do automation by using it.
● Web scraping has multiple uses: From contact scraping and trawling social
media for brand mentions to carrying out SEO audits, the possibilities are
endless.
● Python is a popular tool for scraping the web: Python libraries like
Beautifulsoup, scrapy, and pandas are all common tools for scraping the
web.
● Don’t break the law: Before scraping the web, check the laws in various
jurisdictions, and be mindful not to breach a site’s terms of service.
Types of API’s:
1. Open API’s.
2. Partner API’s.
3. Internal API’s.
4. Composite API’s.
These types of APIs we can use to scrape data and some might be paid and
might not be.
API’s found to extract data:
For facebook, there are three mostly common API’s:
1. Octoparse.
2. Graph API.
3. Visual Scraper API.
After doing the proper research I get to know some rules for scraping:
Like google does not allow scraping as It is possible to scrape the normal
result pages but Google does not allow it. If you scrape at a rate higher than
8 keyword requests per hour you risk detection, higher than 10/h will get
you blocked.
Linkedin also does not allow us to scrape its data. After some tries we get
risk detection and the account gets blocked or banned.
TASK - 2:
Learn How to Use Python Libraries :
The second task was to understand the open source Libraries that will be
used during the application development in python. The Python Standard
Library contains the exact syntax, semantics, and tokens of Python. It
contains built-in modules that provide access to basic system functionality
like I/O and some other core modules.
I get to know the libraries from python which I can use to scrape the data.
The best library I found to scrape the data is the selenium library for good
automation inside python to get to the data and after that the scraping will
start.
“For Learning Purposes Only”
DAY – 3:
A BOT TO SCRAPE DATA
TASK - 1:
Research for a website :
On the first task of day-3, I Researched for a webpage from where i would
be able to get data to scrape as a learner.
After doing a good research for hours, I found a webpage named Clutch
where I can get data which a learner can scrape.
*The data that I will be obtaining, will be used for my learning purposes
only and will be deleted.
Clutch is a B2B research, ratings and reviews site that identifies leading IT
and marketing service providers and software. Clutch evaluates companies
based on over a dozen quantitative and qualitative factors, including client
reviews, company experience, client list, industry recognition, and market
presence. Clutch helps companies manage their online reputation through
3rd party, verified reviews and increases their online visibility and traffic.
Services & Solutions offered by Clutch:
● Advertising & Marketing
● Search Engine Optimization (SEO)
● Mobile App Development
● Web & Software development
● Web Design
● IT Services & Solutions
● Business Services
TASK - 2:
Trying to Scrape Data:
I created a bot in python using different open source libraries of python.
Some of the libraries needed to be installed first before creating the
python application. The libraries are:
1. Beautiful Soup.
2. Requests.
3. lxml.
4. Pandas.
5. openpyxl.
6. selenium.
BeautifulSoup library is used to parse html data. i.e: this library is used
to get html data elements into python directly from the internet. It is used
to navigate, search and modify the parse tree of the DOM.
Request is an elegant and simple HTTP library for Python through which
http requests can be made to a server. This library allows a developer to
make http requests on behalf of the system.
Selenium Library is a web testing library for Robot Framework that utilizes
the Selenium tool internally. The project is hosted on GitHub and downloads
can be found from PyPI. SeleniumLibrary works with Selenium 3 and 4. It
supports Python 3.6 or newer.
To install the Selenium bindings in our system, run the command: pip install
selenium. As this is done, a folder called Selenium should get created within
the Python folder. To update the existing version of Selenium, run the
command: pip install –U selenium.
And thus, using these libraries I can start scraping the data from scratch and
can start developing a program about that.
By the end of the day, I have successfully created a bot that was able to
scrape data from clutch which a learner can access
“For Learning Purposes Only”
DAY – 4:
A BOT TO SCRAPE DATA
TASK - 1:
Write python program from scratch:
The data that will be collected from Clutch will be of web designers from all
location:
we need to make a request object that will request each time a new http
request is made.
i.e:
text=requests.get(“https://clutch.co/web-designers/”).text
the received data will be stored in a variable (text) , we need to pass that data
to BeautifulSoup object so that html data can be parsed easily.
i.e:
after passing the data the data is now parsable easily, using BeautifulSoup’s
inbuilt given methods we can find the element and can get the text data from
it.
And so in our program there are patterns in the URL through which
sometimes pagination can be done easily because the change is only the
page number added at the end of the url.
Matching the patterns and finding the pattern is a difficult task to do for a
whole website and after that also applying the logic regarding how, in
which form the pattern is matched.
Like I have to inspect all the pages, check the matching classes and the
html tags they looped inside and find the proper ID or XPath through
which I can put the respective data in the program changing the particular
brackets.
So as observed in the Clutch url the page attribute was adding up in the
url as pages were changed so to get data from all the pages the logic
needed to be inside of a for loop giving data to a formatted string in
python.
i.e:
for i in r:
text = requests.get('https://clutch.co/web-designers?page={}'.format(i)).text
soup = BeautifulSoup(text, 'lxml')
The extracting data from the web page will be also present inside the for
loop that will work on extracting the data while the page gets updated.
Like this the data from all the pages will be collected , stored in a list,
converted into a panda dataframe and then into an excel file.
Around 55,345 firms were extracted for web development from all around
the globe. It took about 25 minutes to extract the data.
And thus, by the end of the day I successfully created the pagination to crawl
through all the pages in the clutch and scrape the data.
After working hard for 2 days, I found out that this is the old school
method to develop a whole program from scratch, while there are
widely available sources from which I can use already made basic
programs and then do the real work of doing something creative in
relatively less amount of time.
By the end of the day I got to know that there’s a platform by Microsoft
called Power Automate where we can create different kinds of
Automation flows.
DAY – 5:
RESEARCH ON POWER AUTOMATE
Research on Power Automate:
● Stage
● Check condition
● Conditional branch
● Default action
● Wait
● Parallel wait branch
● Create record
● Update record
● Assign record
● Send email
● Start child workflow
● Change status
Another disadvantage with workflows in D365 is it can only work with data
inside of Dynamics and requires data integration for outside data. For
additional functionalities, custom workflow extensions need to be created as
well.
Using power automation reduces app development costs by 70%. Users can
create custom workflows for their organization with little coding technical
knowledge. Workflows in Power Automate can be triggered based on button
push, scheduling date, event, automatic triggers, etc.
There are many types of flows in power automate from which we can create
i.e:
And so by the end of the day, I have a good idea about how microsoft
power automate works.
DAY – 7:
CREATING CLOUD AUTOMATE FLOWS
Cloud Automate that sends emails from excel :
Created a cloud flow that sends emails , the emails are fetched from the
excel column where the receiver's email is present.
The flow created is manually triggered, which means the flow will only run
when the user clicks on run flow.
1. Name.
2. Company.
3. Email.
4. Amount.
5. Amount text.
When a flow will be created the email will be fetched form the email column
of the table.
From the List of rows present in a table we will get the location of our excel
file.
Then we will select an application for all actions in which we will send the
email, our email receiver can be set dynamically by selecting the column
name of the excel from the table.
In the body we can send the message we want to send, in which we can set
the name of the receiver dynamically , where the data will be fetched from
the excel sheet.
DAY – 8:
CREATING CLOUD AUTOMATE FLOWS
Flow to save Gmail attachment into drive:
Created a cloud automate flow that stores the attachment received in the
mail into drive.
This flow is a Manually triggered flow which means this flow can only run
when the user clicks on the run button.
This flow gets the unique user email address received for each email from
different email id’s, then the attachment is fetched from that email and stored
into google drive with the attachment name.
In the above image an email will be sent to the receiver with the subject
“image” and with an image attached name:”jswt lockscreen.png”.
The receiver will receive the email and the attached image, the image will be
directly stored onto the drive’s assigned folder with the given attachment
name.
This flow is a Manually triggered flow which means this flow can only run
when the user clicks on the run button.
This flow gets the unique url to the video when a video is uploaded to the
account.
DAY – 10:
INTRODUCTION TO DESKTOP FLOWS
TASK - 1:
Learn About Desktop Flows:
Desktop flows broaden the existing robotic process automation (RPA)
capabilities in Power Automate and enable you to automate all repetitive
desktop processes. It’s quicker and easier than ever to automate with the new
intuitive Power Automate desktop flow designer using the prebuilt
drag-and-drop actions or recording your own desktop flows to run later.
● Quickly organize your documents using dedicated files and folders actions.
● Accurately extract data from websites and store them in excel files using
Web and Excel automation.
● Apply desktop automation capabilities to put your work on autopilot.
In case you are a home user who is accessing a weather website to see
tomorrow's forecast, or a self-employed businessperson who extracts
information from vendors' invoices or even an employee of a large enterprise
who automates data entry on an ERP system, Power Automate is designed
for you.
TASK - 2:
Create A Desktop Flow:
1. To create a desktop flow in Power Automate, open the app and select New
Flow.
2. Enter a name for the desktop flow, and select OK.
3. Create the flow in the flow designer and press Ctrl+S to save the flow. Close
the flow designer and the flow will appear in the console.
DAY – 11:
MAKING AUTOMATE FLOWS WITH DATAVERSE
Overview Of How To Integrate Flow With Dataverse:
With Microsoft Dataverse, you can store and manage data for business
applications and integrate natively with other Microsoft Power Platform
services like Power BI, Power Apps, Power Virtual Agents, and AI Builder
from your cloud flows.
Use the Microsoft Dataverse connector to create cloud flows that start when
data changes in Dataverse tables and custom messages. For example, you
can send an email whenever a row gets updated in Dataverse.
Overview Of Triggers:
The Microsoft Dataverse connector provides the following triggers to help
you define when your flows start:
● Update a row
● Get a row
● List rows
● Delete a row
● Relate rows
● Unrelated rows
Use business process flows to define a set of steps for people to follow to
take them to a desired outcome. These steps provide a visual indicator that
tells people where they are in the business process. Business process flows
reduce the need for training because new users don’t have to focus on which
table they should be using. They can let the process guide them. You can
configure business process flows to support common sales methodologies
that can help your sales groups achieve better results. For service groups,
business process flows can help new staff get up-to-speed more quickly and
avoid mistakes that could result in unsatisfied customers.
Each stage contains a group of steps. Each step represents a column where
data can be entered. You can advance to the next stage by using the Next
Stage button. In the unified interface, you can work with a business process
flow stage inside the stage flyout or you can pin it to the side pane. Business
process flows don't support expanding the stage flyout to the side pane on
mobile devices.
You can make a step required so that people must enter data for a
corresponding column before they can proceed to the next stage. This is
commonly called ”stage-gating”. If you are adding a business-required or
system-required column to a business process flow stage, we recommend
that you add this column to your form as well.
Business process flows appear relatively simple compared to other types of
processes because they do not provide any conditional business logic or
automation beyond providing the streamlined experience for data entry and
controlling entry into stages. However, when you combine them with other
processes and customizations, they can play an important role in saving
people time, reducing training costs, and increasing user adoption.
When someone creates a new table row, the list of available active business
process definitions is filtered by the user’s security role. The first activated
business process definition available for the user’s security role according to
the process order list is the one applied by default. If more than one active
business process definition is available, users can load another from the
Switch Process dialog. Whenever processes are switched, the one currently
rendered goes to the background and is replaced by the selected one, but it
maintains its state and can be switched back. Each row can have multiple
process instances associated (each for a different business process flow
definition, up to a total of 10). On form load, only one business process flow
is rendered. When any user applies a different process, that process may only
load by default for that particular user.
CONCLUSION :
● Hereby, I have learnt the importance of using low code power automation
processes.
● Successfully created three power automate applications:
d. To learn python :
Python Tutorial (w3schools.com)
THANK YOU!