Scikit Learn 股票投资：p12

▌前言

到目前为止， 我们已经掌握了对单一数据Total Debt/Equity (mrq)的提取和整理。 接下来这个教程主要教大家如何将单一数据扩展为多数据的处理。

▌视频

▌视频出处

视频系列：Scikit-learn Machine Learning with Python and SKlearn

本视频出处：Scikit Learn Machine Learning for investing Tutorial with Python p. 12

哔哩哔哩：Scikit Learn Machine Learning for investing Tutorial with Python p. 12

▌内容

最初我们建立一个function的时候，我们只定义了def Key_Stats(gather="Total Debt/Equity (mrq)"):。 gather= 后面就是填写我们想要获取的数据。 如果想从文件仲获取多项数据，那么我们就需要改写：

def Key_Stats(gather="Total Debt/Equity (mrq)"):

改为：

def Key_Stats(
    gather=[
      "Total Debt/Equity",
      'Trailing P/E',
      'Price/Sales',
      'Price/Book',
      'Profit Margin',
      'Operating Margin',
      'Return on Assets',
      'Return on Equity',
      'Revenue Per Share',
      'Market Cap',
      'Enterprise Value',
      'Forward P/E',
      'PEG Ratio',
      'Enterprise Value/Revenue',
      'Enterprise Value/EBITDA',
      'Revenue',
      'Gross Profit',
      'EBITDA',
      'Net Income Avl to Common ',
      'Diluted EPS',
      'Earnings Growth',
      'Revenue Growth',
      'Total Cash',
      'Total Cash Per Share',
      'Total Debt',
      'Current Ratio',
      'Book Value Per Share',
      'Cash Flow',
      'Beta',
      'Held by Insiders',
      'Held by Institutions',
      'Shares Short (as of',
      'Short Ratio',
      'Short % of Float',
      'Shares Short (prior '
    ]
  ):

接下来，因为现在从提取单项数据变为多项数据，其中有些数据未跟随这B，M或者N/A，我们需要 利用正则表达式替换提取value的部分。

#######源代码#######
 try:
            value = float(source.split(gather+':</td><td class="yfnc_tabledata1">')[1].split('</td>')[0])
          except Exception as e:
            try:             
              value = float(source.split(gather+':</td>\n<td class="yfnc_tabledata1">')[1].split('</td>')[0])
            except Exception as e:
              pass

更改为：

try:
  value_list = []
  #遍历需要提取的数据项gather
  for each_data in gather:
      try:
        regex = re.escape(each_data) + r'.*?(\d{1,8}\.\d{1,8}M?B?|N/A)%?</td>'
        value = re.search(regex, source)
        value = (value.group(1))

        #如果得出的数据为B，那么将其转换为10亿
        if "B" in value:
            value =float(value.replace("B","")) * 1000000000
        #如果得出的数据为M，那么将其转换为百万
        elif "M" in value:
            value = float(value.replace("M", "")) * 1000000
        value_list.append(value)

      except Exception as e:
        value = "N/A"
        value_list.append(value)

上述代码应用了正则表达式，re.escape(each_data) + r'.*?(\d{1,8}\.\d{1,8}M?B?|N/A)%?</td>'。
re.escape 的用法

Escape all the characters in pattern except ASCII letters and numbers. This is useful if you want to match an arbitrary literal string that may have regular expression metacharacters in it. For example:

按照官方的解释，这个函数主要是用于跳过如”$ * . ^”等特殊技字符，只保留字母和数字。

print re.escape('python.exe')
python\.exe

正则表达式：

r'.*?(\d{1,8}\.\d{1,8}M?B?|N/A)%?</td>'

解析:

1. 首先，这是一个字符串，前面的一个 r 表示字符串为非转义的原始字符串，让编译器忽略反斜杠，也就是忽略转义字符。但是这个字符串里没有反斜杠，所以这个 r 可有可无。
2. .*? 代表非贪婪模式，也就是说只匹配符合条件的最少字符
3. \d{1,8}获取数字1-8
4. . 为获取dot
5. \d{1,8}表示.后面继续获取数字1-8
6. M? B? 匹配前一个字符或子表达式0次或1次重复
7. |N/A， “|” 表示 “或者”。 如果出现数据缺失，我们用|N/A来表示。
8. %?， 处理%符号
9. ， 表示以为结尾。

接下来需要处理缺失的数据，因为用Machine Learning的时候需要确保数据的完整性。

if difference > 0:
             status = "outperform"
         else:
             status = "underperform"
         #扔掉缺失的数据。 除了这种方法，我们还可以将缺失数据设为999999，Machine Learning会将outlier自动扔掉。    
         if value_list.count("N/A") > 0:
             pass
         else:
         ..................

然后我们需要更改 df = df.append的代码为：

df = df.append(
{
  'Date':date_stamp,
  'Unix':unix_time,
  'Ticker':ticker,
  'Price':stock_price,
  'stock_p_change':stock_p_change,
  'SP500':sp500_value,
  'sp500_p_change':sp500_p_change,
  'Difference':difference,
  'DE Ratio':value_list[0],
  #'Market Cap':value_list[1],
  'Trailing P/E':value_list[1],
  'Price/Sales':value_list[2],
  'Price/Book':value_list[3],
  'Profit Margin':value_list[4],
  'Operating Margin':value_list[5],
  'Return on Assets':value_list[6],
  'Return on Equity':value_list[7],
  'Revenue Per Share':value_list[8],
  'Market Cap':value_list[9],
  'Enterprise Value':value_list[10],
  'Forward P/E':value_list[11],
  'PEG Ratio':value_list[12],
  'Enterprise Value/Revenue':value_list[13],
  'Enterprise Value/EBITDA':value_list[14],
  'Revenue':value_list[15],
  'Gross Profit':value_list[16],
  'EBITDA':value_list[17],
  'Net Income Avl to Common ':value_list[18],
  'Diluted EPS':value_list[19],
  'Earnings Growth':value_list[20],
  'Revenue Growth':value_list[21],
  'Total Cash':value_list[22],
  'Total Cash Per Share':value_list[23],
  'Total Debt':value_list[24],
  'Current Ratio':value_list[25],
  'Book Value Per Share':value_list[26],
  'Cash Flow':value_list[27],
  'Beta':value_list[28],
  'Held by Insiders':value_list[29],
  'Held by Institutions':value_list[30],
  'Shares Short (as of':value_list[31],
  'Short Ratio':value_list[32],
  'Short % of Float':value_list[33],
  'Shares Short (prior ':value_list[34],
  'Status':status
},
ignore_index=True)

▌源代码

import pandas as pd
import os
import time
from datetime import datetime
import re
from time import mktime
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import style
style.use("dark_background")


#获取数据的具体路径
path = "../intraQuarter"

#定义一个function，默认值为Total Debt/Equity (mrq)，以后可以更改为其他
def Key_Stats(
    gather=[
      "Total Debt/Equity",
      'Trailing P/E',
      'Price/Sales',
      'Price/Book',
      'Profit Margin',
      'Operating Margin',
      'Return on Assets',
      'Return on Equity',
      'Revenue Per Share',
      'Market Cap',
      'Enterprise Value',
      'Forward P/E',
      'PEG Ratio',
      'Enterprise Value/Revenue',
      'Enterprise Value/EBITDA',
      'Revenue',
      'Gross Profit',
      'EBITDA',
      'Net Income Avl to Common ',
      'Diluted EPS',
      'Earnings Growth',
      'Revenue Growth',
      'Total Cash',
      'Total Cash Per Share',
      'Total Debt',
      'Current Ratio',
      'Book Value Per Share',
      'Cash Flow',
      'Beta',
      'Held by Insiders',
      'Held by Institutions',
      'Shares Short (as of',
      'Short Ratio',
      'Short % of Float',
      'Shares Short (prior '
    ]
  ):

  statspath = path+'/_KeyStats'

  #os模块提供的walk方法很强大，能够把给定的目录下的所有目录和文件遍历出来。
  #方法：os.walk(path),遍历path，返回一个对象，他的每个部分都是一个三元组,('目录x'，[目录x下的目录list]，目录x下面的文件)
  stock_list = [x[0] for x in os.walk(statspath)]
  df = pd.DataFrame(
    columns = [
      'Date',
      'Unix',
      'Ticker',
      'Price',
      'stock_p_change',
      'SP500',
      'sp500_p_change',
      'Difference',
      ##############以下为新添加列
      'DE Ratio',
      'Trailing P/E',
      'Price/Sales',
      'Price/Book',
      'Profit Margin',
      'Operating Margin',
      'Return on Assets',
      'Return on Equity',
      'Revenue Per Share',
      'Market Cap',
      'Enterprise Value',
      'Forward P/E',
      'PEG Ratio',
      'Enterprise Value/Revenue',
      'Enterprise Value/EBITDA',
      'Revenue',
      'Gross Profit',
      'EBITDA',
      'Net Income Avl to Common ',
      'Diluted EPS',
      'Earnings Growth',
      'Revenue Growth',
      'Total Cash',
      'Total Cash Per Share',
      'Total Debt',
      'Current Ratio',
      'Book Value Per Share',
      'Cash Flow',
      'Beta',
      'Held by Insiders',
      'Held by Institutions',
      'Shares Short (as of',
      'Short Ratio',
      'Short % of Float',
      'Shares Short (prior ',                                
      ##############
      'Status'
    ]
  )

  #读取SP500指数数据
  sp500_df = pd.DataFrame.from_csv("SPY.csv")

  ticker_list = []

  #stock_list[1:] -- 主要作用是跳过根目录intraQuarter
  for each_dir in stock_list[1:]:
    #os.listdir(each_dir)：列出each_dir下的目录和文件
    each_file = os.listdir(each_dir)

    # ticker = each_dir.split("\\")[1] # Windows only
    # ticker = each_dir.split("/")[1] # this didn't work so do this:
    ticker = os.path.basename(os.path.normpath(each_dir))
    # print(ticker) # 用作验证是否成功获取股票代码
    ticker_list.append(ticker)

    starting_stock_value = False
    starting_sp500_value = False

    if len(each_file) > 0:
      for file in each_file:
        #将文件名转换为时间序列
        date_stamp = datetime.strptime(file, '%Y%m%d%H%M%S.html')
        #转换为unix_time
        unix_time = time.mktime(date_stamp.timetuple())
        full_file_path = each_dir+'/'+file
        #读取html文件
        source = open(full_file_path,'r').read()
        try:
          value_list = []

          for each_data in gather:
              try:
                regex = re.escape(each_data) + r'.*?(\d{1,8}\.\d{1,8}M?B?|N/A)%?</td>'
                value = re.search(regex, source)
                value = (value.group(1))

                if "B" in value:
                    value =float(value.replace("B","")) * 1000000000
                elif "M" in value:
                    value = float(value.replace("M", "")) * 1000000
                value_list.append(value)

              except Exception as e:
                value = "N/A"
                value_list.append(value)

          try:
            sp500_date = datetime.fromtimestamp(unix_time).strftime('%Y-%m-%d')
            row = sp500_df[(sp500_df.index == sp500_date)]
            sp500_value = float(row["Adj Close"])
          except:
            sp500_date = datetime.fromtimestamp(unix_time-259200).strftime('%Y-%m-%d')
            row = sp500_df[(sp500_df.index == sp500_date)]
            sp500_value = float(row["Adj Close"])

          try:
             #将文件内容以':</td><td class="yfnc_tabledata1">'开始进行分割，[1]表示分割后获取后面的第一个内容D/E的值
            stock_price = float(source.split('</small><big><b>')[1].split('</b></big>')[0])
          except Exception as e:
            #    <span id="yfs_l10_afl">43.27</span>
            try:
              #因为yahoo中的数据经过调整，获取页面数值的位置已经更改，所以要利用正则表达式通过re.search查找出数值
              #r'(\d{1,8}\.\d{1,8})'中的\d表示数字，{1,8}表示查找1-8数字，中间的\.表示跳过数值中的"."
              stock_price = (source.split('</small><big><b>')[1].split('</b></big>')[0])
              stock_price = re.search(r'(\d{1,8}\.\d{1,8})',stock_price)
              stock_price = float(stock_price.group(1))
              #print(stock_price)
            except Exception as e:
              try:
                #原因同上
                stock_price = (source.split('<span class="time_rtq_ticker">')[1].split('</span>')[0])
                stock_price = re.search(r'(\d{1,8}\.\d{1,8})',stock_price)
                stock_price = float(stock_price.group(1))
              except Exception as e:
                print(str(e),'stock_price 3rd try+except',file,ticker)

              #print('Latest:',stock_price)
              #print('stock price',str(e),ticker,file)
              #time.sleep(15)

          #print("stock_price:",stock_price,"ticker:", ticker)

          if not starting_stock_value:
            starting_stock_value = stock_price
          if not starting_sp500_value:
            starting_sp500_value = sp500_value

          stock_p_change = ((stock_price - starting_stock_value) / starting_stock_value) * 100
          sp500_p_change = ((sp500_value - starting_sp500_value) / starting_sp500_value) * 100
          difference = stock_p_change - sp500_p_change

          if difference > 0:
              status = "outperform"
          else:
              status = "underperform"

          if value_list.count("N/A") > 0:
              pass

          else:
              #将数据叠加并保存在字典里面
              df = df.append(
              {
                'Date':date_stamp,
                'Unix':unix_time,
                'Ticker':ticker,
                'Price':stock_price,
                'stock_p_change':stock_p_change,
                'SP500':sp500_value,
                'sp500_p_change':sp500_p_change,
                'Difference':difference,
                'DE Ratio':value_list[0],
                #'Market Cap':value_list[1],
                'Trailing P/E':value_list[1],
                'Price/Sales':value_list[2],
                'Price/Book':value_list[3],
                'Profit Margin':value_list[4],
                'Operating Margin':value_list[5],
                'Return on Assets':value_list[6],
                'Return on Equity':value_list[7],
                'Revenue Per Share':value_list[8],
                'Market Cap':value_list[9],
                'Enterprise Value':value_list[10],
                'Forward P/E':value_list[11],
                'PEG Ratio':value_list[12],
                'Enterprise Value/Revenue':value_list[13],
                'Enterprise Value/EBITDA':value_list[14],
                'Revenue':value_list[15],
                'Gross Profit':value_list[16],
                'EBITDA':value_list[17],
                'Net Income Avl to Common ':value_list[18],
                'Diluted EPS':value_list[19],
                'Earnings Growth':value_list[20],
                'Revenue Growth':value_list[21],
                'Total Cash':value_list[22],
                'Total Cash Per Share':value_list[23],
                'Total Debt':value_list[24],
                'Current Ratio':value_list[25],
                'Book Value Per Share':value_list[26],
                'Cash Flow':value_list[27],
                'Beta':value_list[28],
                'Held by Insiders':value_list[29],
                'Held by Institutions':value_list[30],
                'Shares Short (as of':value_list[31],
                'Short Ratio':value_list[32],
                'Short % of Float':value_list[33],
                'Shares Short (prior ':value_list[34],
                'Status':status
              },
              ignore_index=True)

        except Exception as e:
          pass

  df.to_csv('key_Stats.csv')

Key_Stats()

▌最后

虽然分c君_BingWong只是作为一名搬运工，连码农都称不上。 但制作代码中的注释、翻译和搬运都花了很多时间，请各位大侠高抬贵手，在转载时请注明出处。